Re: 'make depend' or 'make' bug on recent --current

On 6/1/16 11:49 AM, Andrey Chernov wrote:
> On 01.06.2016 21:18, Bryan Drewery wrote:
>> On 6/1/2016 6:11 AM, Andrey Chernov wrote:
>>> Steps to reproduce:
>>>
>>> cd /usr/src/lib/libc/stdlib
>>> touch *div*.c
>>> cd ..
>>> make depend
>>> make
>>>
>>> And see how imaxdiv.o only is recompiled.
>>> No div.o ldiv.o lldiv.o are recompiled.

Because they never were compiled!

lib/libc/stdlib/Makefile.inc
MISRCS has div.c ldiv.c lldiv.c

However, for at least amd64 it also includes:
lib/libc/amd64/stdlib/Makefile.inc
which has MDSRCS= div.S ldiv.S lldiv.S

Note the .depend.lldiv.o file:
> ~/git/freebsd/lib/libc/stdlib # cat /usr/obj/root/git/freebsd/lib/libc/.depend.lldiv.o
> lldiv.o: /root/git/freebsd/lib/libc/amd64/stdlib/lldiv.S \
>   /usr/obj/root/git/freebsd/tmp/usr/include/machine/asm.h \
>   /usr/obj/root/git/freebsd/tmp/usr/include/sys/cdefs.h


It's built from lldiv.S, not lldiv.c. No bug here.

proof:

> cc -O2 -pipe   -I/root/git/freebsd/lib/libc/include -I/root/git/freebsd/lib/libc/../../include -I/root/git/freebsd/lib/libc/amd64 -DNLS  -D__DBINTERFACE_PRIVATE -I/root/git/freebsd/lib/libc/../../contrib/
> gdtoa -I/root/git/freebsd/lib/libc/../../contrib/libc-vis -DINET6 -I/usr/obj/root/git/freebsd/lib/libc -I/root/git/freebsd/lib/libc/resolv -D_ACL_PRIVATE -DPOSIX_MISTAKE -I/root/git/freebsd/lib/libc/../li
> bmd -I/root/git/freebsd/lib/libc/../../contrib/jemalloc/include -DMALLOC_PRODUCTION -I/root/git/freebsd/lib/libc/../../contrib/tzcode/stdtime -I/root/git/freebsd/lib/libc/stdtime -I/root/git/freebsd/lib/l
> ibc/locale -DBROKEN_DES -DPORTMAP -DDES_BUILTIN -I/root/git/freebsd/lib/libc/rpc -DYP -DNS_CACHING -DSYMBOL_VERSIONING -MD  -MF.depend.lldiv.o -MTlldiv.o -std=gnu99 -fstack-protector-strong -Wsystem-heade
> rs -Werror -Wall -Wno-format-y2k -Wno-uninitialized -Wno-pointer-sign -Wno-empty-body -Wno-string-plus-int -Wno-unused-const-variable -Wno-tautological-compare -Wno-unused-value -Wno-parentheses-equality
> -Wno-unused-function -Wno-enum-conversion -Wno-unused-local-typedef -Wno-switch -Wno-switch-enum -Wno-knr-promoted-parameter  -fcolor-diagnostics -Qunused-arguments  -I/root/git/freebsd/lib/libutil -I/roo
> t/git/freebsd/lib/msun/amd64 -I/root/git/freebsd/lib/msun/x86 -I/root/git/freebsd/lib/msun/src   -c /root/git/freebsd/lib/libc/amd64/stdlib/lldiv.S -o lldiv.o


> cc -fpic -DPIC -O2 -pipe   -I/root/git/freebsd/lib/libc/include -I/root/git/freebsd/lib/libc/../../include -I/root/git/freebsd/lib/libc/amd64 -DNLS  -D__DBINTERFACE_PRIVATE -I/root/git/freebsd/lib/libc/..
> /../contrib/gdtoa -I/root/git/freebsd/lib/libc/../../contrib/libc-vis -DINET6 -I/usr/obj/root/git/freebsd/lib/libc -I/root/git/freebsd/lib/libc/resolv -D_ACL_PRIVATE -DPOSIX_MISTAKE -I/root/git/freebsd/li
> b/libc/../libmd -I/root/git/freebsd/lib/libc/../../contrib/jemalloc/include -DMALLOC_PRODUCTION -I/root/git/freebsd/lib/libc/../../contrib/tzcode/stdtime -I/root/git/freebsd/lib/libc/stdtime -I/root/git/f
> reebsd/lib/libc/locale -DBROKEN_DES -DPORTMAP -DDES_BUILTIN -I/root/git/freebsd/lib/libc/rpc -DYP -DNS_CACHING -DSYMBOL_VERSIONING -MD  -MF.depend.lldiv.So -MTlldiv.So -std=gnu99 -fstack-protector-strong
> -Wsystem-headers -Werror -Wall -Wno-format-y2k -Wno-uninitialized -Wno-pointer-sign -Wno-empty-body -Wno-string-plus-int -Wno-unused-const-variable -Wno-tautological-compare -Wno-unused-value -Wno-parenth
> eses-equality -Wno-unused-function -Wno-enum-conversion -Wno-unused-local-typedef -Wno-switch -Wno-switch-enum -Wno-knr-promoted-parameter  -fcolor-diagnostics -Qunused-arguments  -I/root/git/freebsd/lib/
> libutil -I/root/git/freebsd/lib/msun/amd64 -I/root/git/freebsd/lib/msun/x86 -I/root/git/freebsd/lib/msun/src    -c /root/git/freebsd/lib/libc/amd64/stdlib/lldiv.S -o lldiv.So

note lldiv.S in both.

>>
>> My dev system is busy at the moment. I'll test it and get back to you.
> 
> I need to add that I do 'make cleandepend/cleandir/cleanobj' + 'make
> obj' again and full rebuild with no old files, but the bug repeated again.
> 
>>> P.S. new make depend is simple disgusting. It tends to recompile
>>> everything in the system if some minor header file is touched, but
>>
>> If the header is used by all source files then that is expected.
>>
>> However if you do not have a .depend.obj.o file then it is quite
>> aggressive with building.  If you touch any header it will rebuild
>> everything.  But you shouldn't get into that situation unless you rm -f
>> .depend* first.
>>
>>> completely forget to recompile source code changes. I suggest to back
>>> out all AI in that area.
>>> 'make depend' is not time-consuming task and good old way never made
>>> mistakes.
>>
>> The graph in the original commit for WITH_FAST_DEPEND disagrees.
>> https://svnweb.freebsd.org/base?view=revision&revision=290433
>>
>> We run the preprocessor once now, not twice.
> 
> It sounds good, just implemented bad. You measure some spherical chicken
> in vacuum, not what really happens. In the old times I almost never have
> clang libs rebuild (few files from there max when FreeBSD_version is
> increased), but now I got them fully rebuilt with any header change.
> That is the biggest slowdown and not what you try to measure.
> Don't ever use 'make world'. Try to rebuild the system incrementally and
> you'll see.

I cannot argue with a lack of solid evidence.

The old 'make depend' ran 'mkdep' which ran 'cc -E -M' which produces
*the same output* as 'cc foo.c -o foo.o -MD -MF .depend.foo.o -MT
foo.o'.  There's nothing different in the actual .depend file
implementation/content.  Clang rebuilds often because it is changing
often!  Just look at recent commit logs and you'll see r300984 which
will cause a rebuild of clang.  Or r301011 which modified
sys/sys/param.h which will rebuild just about everything.  These are
normal and how the old system worked as well.

There certainly are some issues with the new system.
1. Processing the split files can be slow over NFS
2. make cleandepend - will cause the next build to make a lot of guesses
and not be very efficient.
3. make cleandepend - There is no way to generate the .depend* files
again without rebuilding.
4. It doesn't fix all of the missing dependencies that have been missing
forever such as crt, csu, libgcc, etc for static builds.
5. the csu builds don't use it yet but have workarounds for it
6. removing the 'make depend' tree-walk can hurt downstream builds which
relied on having a multi-phase build to generate a .mk file to include
(odd case I hit at work).  No such case exists in the upstream FreeBSD tree.

-- 
Regards,
Bryan Drewery