Re: [TEST] make -j patch [take 2]

Hi Poul-Henning,

Would you please put yourself into constructive mode now?  ;)

On Fri, Nov 12, 2004 at 09:09:13AM +0100, Poul-Henning Kamp wrote:
> In message <20041112080416.GA41844_at_ip.net.ua>, Ruslan Ermilov writes:
> 
> >> Here is take two of my "make -j" patch.  Further testing found
> >> a couple of buglets.  If this survices further testing, it will
> >> be committed in a couple of days.
> >>
> >> With this patch "make -j N" will put the load average as close
> >> to N as the makefiles will allow.
> >>
> >This will make it impossible for "make universe" to build four
> >worlds in parallel, each world with 8 jobs.  It's possible now,
> >please see the commit log for src/Makefile,v 1.308 for details.
> 
> Yes, isn't it wonderful ?  We can get a way from that hack and
> have make actually do global resource management:
> 
> I ran "make -j 12 universe" overnight and the loadaverage danced
> between 10 and 13 most of the time, with a spike at 14 while
> cron did its late-night raid-the-fridge thing.
> 
> Last I tried this without my patch, the loadaverage spiked at 65.
> 
But you don't give an opportunity to control this on a sub-make
level (that's what I ask for).  If you implement what I suggest,
"make -j 12 universe" will still do as you saw it last night,
while still allowing "make -j4 universe JFLAG=-j3" to build four
worlds in parallel, each with three jobs.  Isn't it nice?  ;)

I also have a question.  What will happen (with your patch) if
make was given -j4, and then I give a sub-make an explicit
argument of -j8?  I think it will cause the main "make" to
behave like -j1, supposing there're a lot of jobs in sub-make
and they're completed and started quickly.  Yes/no?

> >Please abandon your bsd.subdir.mk patch completely.  ;)
> 
> There is something one of us do not understand because that patch
> gives me measurably more parallelism.
> 
I've spent some time explaining why your patch doesn't add any
parallelism to bsd.subdir.mk, and why parallelism in bsd.subdir.mk
should not be allowed.   Did you spend your time trying to
understand it?  ;)

If you want a quick prove, please try these with the attached tar:

	make -j2 all
	make -j2 par-all

with or withour your patch to bsd.subdir.mk applied, doesn't
matter.  If SUBDIRs are parallelized, then "b" will be printed
before "a".  The par-all demonstrates how does this work.
The "make -j2 all" should prove you that your patch doesn't
parallelize SUBDIR.  But more important is that _SUBDIR macro
should NOT be parallelized, as it will break a lot of things.
The fact that you tried your patch to bsd.subdir.mk with -jX
and it didn't break build is another proof that it didn't
parallelize SUBDIRs.

Or perhaps you could explain how your patch helps parallelism
or improves something, I fail to see it.  All I see is that
it causes N shells to be used instead of one, but this is
unmeasureable in buildworld times.


Cheers,
-- 
Ruslan Ermilov
ru_at_FreeBSD.org
FreeBSD committer