Re: [TEST] make -j patch [take 2]

Hi Poul-Henning,

On Fri, Nov 12, 2004 at 12:36:53AM +0100, Poul-Henning Kamp wrote:
> In message <80546.1100202141_at_critter.freebsd.dk>, Poul-Henning Kamp writes:
> 
> Here is take two of my "make -j" patch.  Further testing found
> a couple of buglets.  If this survices further testing, it will
> be committed in a couple of days.
> 
> With this patch "make -j N" will put the load average as close
> to N as the makefiles will allow.
> 
This will make it impossible for "make universe" to build four
worlds in parallel, each world with 8 jobs.  It's possible now,
please see the commit log for src/Makefile,v 1.308 for details.

It would be great if -j specified on the command line (rather
than derived from MAKEFLAGS environment variable on startup)
would cause make(1) to "forget" its group membership regarding
parallelism.  This can be implemented as simple as unsetting
the MAKE_JOBS_FIFO environment variable if -j was specified
on the command line.  If this is done, I'll be quite happy to
see this committed, otherwise not.  FWIW.  ;)

> Included is also a patch I've been using to make the SUBDIR targets
> go parallel.  Ruslan will have to fix all the mistakes I've made
> in that one before it gets committed.
> 
SUBDIR targets should NOT be built in parallel; there are a LOT
of places in our world that assume some order (let me know if
you need some examples).  For this whole reason, the _SUBDIR
macro is constructed so that it cannot be parallelized.

Your patch to bsd.subdir.mk doesn't change it in any good way.
I mean, using .for loops doesn't make it any more parallel.
One should think of

	.for foo in a b
	do ${foo}
	.endfor

as

	do a
	do b

That's how make(1) internally parses it, and the latter in NOT
parallelizeable.  To make something parallel, you should split
the job into multiple targets, for example:

all: do_a do_b
.for t in a b
do_${t}:
	do ${foo}
.endfor

Then do_a and do_b can be built in parallel, unless told otherwise
by the .ORDER directive.  This is what the last chunk of
Makefile.inc1 does for par-* targets: SUBDIRs will be built in
parallel in Makefile.inc1 (sometimes, when it's safe).

Your patch replaces the shell loop with an equivalent make loop,
causing N shells to be used instead of only one, where N is the
SUBDIR list length.  This is not good.  It also adds some style
and functional bugs, like loosing the ability for "make -nn"
(double-n) to descend into subdirs.  This is quite nice that
we have now (today, one can "make -nn buildworld" and see what
happens).

Please abandon your bsd.subdir.mk patch completely.  ;)


Cheers,
-- 
Ruslan Ermilov
ru_at_FreeBSD.org
FreeBSD committer