Re: My build work and goals

Firstly, nice writup - a few extra blank lines might have helped my eyes
though.

Bryan Drewery <bdrewery_at_freebsd.org> wrote:
> This mail is to outline my recent work, goals and motivations.  This is
> long.  This is not really a architectural review or discussion mail.  I
...

> Some problems with the combined DIRDEPS_BUILD build:
> 
> It is very hard to maintain, prone to massive churn in directories you
> did and did not touch, requires checking in fickle generated files, has
> a chicken-and-egg problem for adding new things, does not respect
> WITH/WITHOUT options or ARCH-dependent SUBDIR, has subtle problems with
> a static dependencies list around options, and has no 'installworld'
> support.

Dealing with all the optional cases is definitely a problem to be
solved.
I've a few ideas on that front, but haven't had time to experiment.

FWIW we (Juniper) see zero "churn" in our dependencies
a/ we use a fixed set of options
b/ we've fixed the bugs in makefiles that were causing churn.

Ignoring the WITH/WITHOUT options stuff for a sec, dependency churn is
usually a result of bugs in the leaf makefiles.
These are often easy to spot, sometimes quite hard.

The bootstrap issue and even just the need to checkin "generated" files
is a head scratcher for a surprising number of people - even after 4+
years of building this way.

> The lack of 'installworld' is a deal breaker, albeit probably trivial to
> fix.  The reason for this is that the system is used to generate
> packages (not pkgng style) at Juniper.  We have our own packaging effort
> occurring in projects/release-pkg though and won't likely use the same
> methods that the DIRDEPS_BUILD does (of which none of the support is
> really in the src tree).

Though AFAIK all the tools we use are, so its really just a matter of
adding some suitable makefiles and manifest - to say build a bootable vm
image - I probably would have done that by now if bhyve supported vmdk
or qcow2 ;-)

But that's likely orthogonal to your point.
Some form of "installworld" is probably needed.

> This build does not use recursive SUBDIR walking (by design).  It uses a
> static list of directories to build in the
> targets/pseudo/userland/*/Makefile.depend files.

That's mostly just for the purpose of demonstration/example.

If/when FreeBSD has some form of manifest/list for what goes into its
"packages" (of whatever form they might be,) then the necessary data
to drive what needs to be built could be obtained from them - as we do
in the Junos build.

> The checked-in Makefile.depend files have a flaw (feature for static
> builds perhaps) that the invoking the linker will cause non-direct
> dependencies to show up in Makefile.depend.  This leads to situations
> such as https://svnweb.freebsd.org/changeset/base/291558.  There is also
> a situation where a local build may set MK_SSP=no and yet need to build
> a dependency that wants MK_SSP=yes (libssp dependency) but because of

Yes, options are a pain.
You can either hard code them via local.dirdeps.mk but it is probably
better to do something a bit like what the current intermediate
makefiles do (one of the idea I mentioned above).
Eg. if the captured dependencies had something like ${lib_libssp} rather than
lib/libssp. If MK_SSP=no it expands to nothing.
I can elaborate in separate thread if you prefer.

> than upstream, is an exhaustive chicken-and-egg problem.  These problems
> make checked-in Makefile.depend infeasible.  "But wait, can you just not
> check them in?"  Not really.

You can, but you lose virtually all the benefits.

> recently added support to local.dirdeps.mk to make an educated guess on
> what DIRDEPS should be even if there is no Makefile.depend.  This is
> based on what is being built (C and C++ objects have common
> dependencies) and what DPADD/LIBADD there are.  In most cases this is
> enough.  It has greatly helped when bootstrapping Makefile.depend.

You still have the issue of local depenenencies.
Sure, only a small percentage of makefiles have problems in this regard,
but humans are generally bad at expressing the dependencies that result
from using things like yacc - get it wrong about 80% of the time.

> Going further though I have written a script that just recursively walks
> the tree to generate the DIRDEPS graph from the first make process
> rather than including the Makefile.depend files.  This removes the need

That's what we used to do, but was deemed too expensive.
Using that to generate Makefile.depend though for your tree, could save
you some overhead on subsequent builds, while allowing tailorying tree
to your set of options.

> P.S. Working on this stuff can be exhausting.  Mistakes are easy.

Thanks for your efforts btw.

--sjg