Re: [TEST] make -j patch [take 2]

In message <20041112090905.GD41844_at_ip.net.ua>, Ruslan Ermilov writes:

>But you don't give an opportunity to control this on a sub-make
>level (that's what I ask for). 

Why would that be of any use ?  If you run "make universe" the task
at hand is to get "make universe" to complete.  You should not care
which partcular submake starts how many jobs when, you should only
care that it works as efficient as possible.

>level (that's what I ask for).  If you implement what I suggest,
>"make -j 12 universe" will still do as you saw it last night,
>while still allowing "make -j4 universe JFLAG=3D-j3" to build four
>worlds in parallel, each with three jobs.  Isn't it nice?  ;)

You're wrong about how this works.

Try plotting your load-avg over time and you will see that you have
neither enforced the 4 nor the 3 jobs.

Or look at the one I just posted:

	http://phk.freebsd.dk/misc/make.png

Then explain to me how you think it is beneficial that "-j 12"
gives make(1) permission to start 65 jobs.

When I say "make -j 12" I mean "12 jobs, damnit!", not "up to
12^(number of levels of subdirectories)".

>I also have a question.  What will happen (with your patch) if
>make was given -j4, and then I give a sub-make an explicit
>argument of -j8?  I think it will cause the main "make" to
>behave like -j1, supposing there're a lot of jobs in sub-make
>and they're completed and started quickly.  Yes/no?

The main make controls.  If you start the main make with -j 4
a global pool of 4 jobs are available to all the submakes
spawned as part of that job.  The submakes will not respect
an explicit "-j N" argument if they find the environment
variable.  You can of course eliminate that if you want to
override this behaviour.

>I've spent some time explaining why your patch doesn't add any
>parallelism to bsd.subdir.mk, and why parallelism in bsd.subdir.mk
>should not be allowed.   Did you spend your time trying to
>understand it?  ;)

When theory and measurement disagree, either can have a problem,
the main difference in approach is that theory has got little to
back up it's claims and must resort to prove experimental error to
survive.

>Or perhaps you could explain how your patch helps parallelism
>or improves something, I fail to see it.  All I see is that
>it causes N shells to be used instead of one, but this is
>unmeasureable in buildworld times.

I don't know the reason, but the result was about 5% speedup on
parallel buildworld.

I'll leave it to you if anything needs done, You're the
makefile wizard.  I've done the bit I set out to do: prevent
boundless creation of jobs.

Poul-Henning

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk_at_FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.