Re: [TEST] make -j patch [take 2]

On Fri, Nov 12, 2004 at 01:11:56PM +0100, Poul-Henning Kamp wrote:
> I have explained this several times already:  You should just leave
> the submakes on their own and let them balance as they see fit.
> 
> I spent a lot of time on parallel/clustered make for a customer and
> the outcome of that is very clear:  Leave the stuff alone after you
> have said your overall intention.
> 
> Obviously you are not going to trust me on this, since you seem to
> have very strong ideas on how you think it works in practice, even
> though you don't seem to have any actual data to back it up with.
> 
> I suggest that you actually try out your ideas in _practice_, not
> just with a few "proof of concept" makefiles made for the purpuse,
> but try to actually _run_ it on real world jobs for some weeks while
> you carefully gater statistics and examine the interactions between
> the parameters given, where given, resulting time to build and
> loading of the machine(s).
> 
> At the very least, do not commit your patch until you have managed
> to come up with at least one instance of real world data where it
> is a good idea.
> 
I gave you one example already, and Harti gave you another.

"make universe" builds 6 worlds today.  By starting it with
"make -j12", you build 6 worlds in parallel, and another 6
jobs are made available for building these worlds.  When one
world finishes, 7 jobs will be available for building 5 other
worlds, etc.  This is fine as long as you're interested in
the final result of "make universe".

When testing a commit candidate change with "make universe",
I want to detect the breakage as early as possible.  This can
be achieved by building less worlds in parallel, and giving
more parallelism to individual world builds.  The overall
time of "make universe" won't change significantly.

If you're by chance familiar with how you can ask Oracle to
optimize your queries, you can either optimize for ALL ROWS or
for FIRST ROWS, and it's up to you to decide on this.  The
"rows" in "make universe" are results of doing buildworld
and buildkernel, and you have control over this.

Do you disagree with anything of the above?

But since resetting MAKE_JOBS_FIFO before starting a sub-make
will have a similar effect, I won't be committing my patch to
make(1).

Can you agree to the below patch?  It doesn't change anything
except when JFLAG is set explicitly to request another level
of parallelism for building individual worlds and kernels?

%%%
Index: Makefile
===================================================================
RCS file: /home/ncvs/src/Makefile,v
retrieving revision 1.310
diff -u -r1.310 Makefile
--- Makefile	6 Nov 2004 03:14:26 -0000	1.310
+++ Makefile	12 Nov 2004 12:58:33 -0000
_at__at_ -252,6 +252,11 _at__at_
 # with a reasonable chance of success, regardless of how old your
 # existing system is.
 #
+.if defined(JFLAG) && !empty(JFLAG)
+UMAKE=	unset MAKE_JOBS_FIFO && ${MAKE} ${JFLAG}
+.else
+UMAKE=	${MAKE}
+.endif
 universe: universe_prologue
 universe_prologue:
 	_at_echo "--------------------------------------------------------------"
_at__at_ -264,7 +269,7 _at__at_
 .ORDER: universe_prologue universe_${mach} universe_epilogue
 universe_${mach}:
 	_at_echo ">> ${mach} started on `LC_ALL=C date`"
-	-cd ${.CURDIR} && ${MAKE} ${JFLAG} buildworld \
+	-cd ${.CURDIR} && ${UMAKE} buildworld \
 	    TARGET_ARCH=${arch} TARGET=${mach} \
 	    __MAKE_CONF=/dev/null \
 	    > _.${mach}.buildworld 2>&1
_at__at_ -290,7 +295,7 _at__at_
 		find [A-Z]*[A-Z] -type f -maxdepth 0 ! -name NOTES
 buildkernels:
 .for kernel in ${KERNCONFS}
-	-cd ${.CURDIR} && ${MAKE} ${JFLAG} buildkernel \
+	-cd ${.CURDIR} && ${UMAKE} buildkernel \
 	    KERNCONF=${kernel} \
 	    __MAKE_CONF=/dev/null \
 	    > _.${TARGET}.${kernel} 2>&1
%%%


Cheers,
-- 
Ruslan Ermilov
ru_at_FreeBSD.org
FreeBSD committer