Re: FYI: Why META_MODE rebuilds so much for building again after installworld (no source changes)

From: Mark Millard <marklmi_at_yahoo.com>
Date: Sat, 23 Jan 2021 21:37:19 -0800
On 2021-Jan-22, at 01:45, Mark Millard <marklmi at yahoo.com> wrote:

> Given an already built, installed and booted system version, I've
> noted a big difference for META_MODE in 2 rebuild contexts (no
> source updates involved):
> 
> *) Prior buildworld buildkernel, installkernel, installworld, boot
>   presumed before (A) and before (B) below.
> 
> A) make . . . buildworld buildkernel
>   make . . . buildworld buildkernel
>   (the 2nd buildworld buildkernel in (A) builds far less than the first)
>   (that means that the first built more than I would have guessed)
> 
> vs.
> 
> B) make . . . buildworld buildkernel
>   make . . . installworld
>   make . . . buildworld buildkernel
>   (the 2nd buildworld buildkernel in (B) builds far more than it did in (A))
>   (so, more like the 1st buildworld buildkernel in (A) and (B), given
>    the specified prior context)
> 
> So I used make -dM for the commented buildworld buildkernel lines, logging
> the build output and later diff'ing them.
> 
> Result that I noticed? Lots of lines uniquely from (B)'s case, ending with
> one of:
> 
> file '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/awk' is newer than the target...
> file '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/cap_mkdb' is newer than the target...
> file '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/cat' is newer than the target...
> file '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/cp' is newer than the target...
> file '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/crunchgen' is newer than the target...
> file '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/crunchide' is newer than the target...
> file '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/dd' is newer than the target...
> file '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/egrep' is newer than the target...
> file '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/env' is newer than the target...
> file '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/file2c' is newer than the target...
> file '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/gencat' is newer than the target...
> file '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/grep' is newer than the target...
> file '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/gzip' is newer than the target...
> file '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/jot' is newer than the target...
> file '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/lex' is newer than the target...
> file '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/ln' is newer than the target...
> file '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/m4' is newer than the target...
> file '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/mv' is newer than the target...
> file '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/patch' is newer than the target...
> file '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/rm' is newer than the target...
> file '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/sed' is newer than the target...
> file '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/sh' is newer than the target...
> file '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/touch' is newer than the target...
> file '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/truncate' is newer than the target...
> file '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/uudecode' is newer than the target...
> file '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/uuencode' is newer than the target...
> file '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/xargs' is newer than the target...
> 
> The lines with these lead to more files being updated and so causing more
> indirect rebuild activity (that cascades).
> 
> Many/most/all(?) of these seem to me to be unlikely to actually need to
> contribute to what needs to be rebuilt (just based on being newer). So
> the option to ignore (some of?) them could be useful in making META_MODE
> builds quicker.

The following from one of the .meta files makes the point that rm use
in the example is unlikely to be important to needing to rebuild,
despite it actually causing a file rebuild. Nor is the specific echo
command listed relevant. Only the "ar" command is:

# Meta data file /usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/lib/libc++/libc++.a.meta
CMD _at_echo building static c++ library
CMD _at_rm -f libc++.a
CMD ar -crsD libc++.a algorithm.o any.o atomic.o barrier.o bind.o charconv.o chrono.o condition_variable.o condition_variable_destructor.o debug.o exception.o filesystem/directory_iterator.o filesyste
m/int128_builtins.o filesystem/operations.o functional.o future.o hash.o ios.o iostream.o locale.o memory.o mutex.o mutex_destructor.o new.o optional.o random.o random_shuffle.o regex.o shared_mutex.o
 stdexcept.o string.o strstream.o system_error.o thread.o typeinfo.o utility.o valarray.o variant.o vector.o cxxrt_auxhelper.o cxxrt_dynamic_cast.o cxxrt_exception.o cxxrt_guard.o cxxrt_libelftc_dem_g
nu3.o cxxrt_memory.o cxxrt_stdexcept.o cxxrt_terminate.o cxxrt_typeinfo.o
CWD /usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/lib/libc++
TARGET libc++.a
-- command output --
building static c++ library

-- filemon acquired metadata --
# filemon version 5
# Target pid 22471
# Start 1611359217.214996
V 5
E 22961 /bin/sh
R 22961 /etc/libmap.conf
R 22961 /var/run/ld-elf.so.hints
R 22961 /lib/libedit.so.7
R 22961 /lib/libc.so.7
R 22961 /lib/libncursesw.so.9
R 22961 /usr/share/locale/C.UTF-8/LC_CTYPE
F 22961 22962
E 22962 /usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/usr/sbin/rm
R 22962 /etc/libmap.conf
R 22962 /var/run/ld-elf.so.hints
R 22962 /lib/libc.so.7
R 22962 /usr/share/locale/C.UTF-8/LC_CTYPE
D 22962 libc++.a
X 22962 0 0
. . .

The timestamp on . . ./tmp/legacy/usr/sbin/rm is not
actually relevant to if libc++.a needs to be rebuilt.

Of course, the structure also point out the judgment
is specific to understanding the sequence of CMD's
listed above. Only a hack of ignoring, not recording,
or commenting out the filemon lines ending in
/tmp/legacy/usr/sbin/rm would seem to avoid the _at_rm
handling issue. Such might well have its own risks.

Some other /tmp/legacy/usr/sbin/* filemon line endings
likely would have a similar status of being a reference
to a file that could(/should?) have its timestamp
relationship not checked.

===
Mark Millard
marklmi at yahoo.com
( dsl-only.net went
away in early 2018-Mar)
Received on Sun Jan 24 2021 - 04:37:27 UTC

This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:41:26 UTC