Re: [TEST] make -j patch [take 2]

On Fri, Nov 12, 2004 at 03:52:37PM +0100, Poul-Henning Kamp wrote:
> In message <20041112130548.GA86162_at_ip.net.ua>, Ruslan Ermilov writes:
> 
> >> At the very least, do not commit your patch until you have managed
> >> to come up with at least one instance of real world data where it
> >> is a good idea.
> >>
> >I gave you one example already, and Harti gave you another.
> >
> >"make universe" builds 6 worlds today.  By starting it with
> >"make -j12", you build 6 worlds in parallel, and another 6
> >jobs are made available for building these worlds.  When one
> >world finishes, 7 jobs will be available for building 5 other
> >worlds, etc.  This is fine as long as you're interested in
> >the final result of "make universe".
> 
> You have not actually _tried_ running make universe with
> my patch, have you ?
> 
> I thought so :-(
> 
> make -j 12 universe will make 12 jobs available.  The main
> make will grab six of these to start the buildworlds, and
> those six buildworlds compete for the remaining 6 tokens.
> 
> The result is that those six buildworlds progress _in parallel_
> each of them using as many tokens as they can lay their hands on
> at any one time.
> 
This is exactly what I said above.  I well understand that
six buildworlds will compete for six remaining slots, then
five buildworld will compete for seven slots, and so on.

> If you want them to have more slots, say "-j 18" instead
> and they will have 12 slots to compete for.
> 
> This is, as far as I can tell, just the thing you want to happen,
> only it doesn't suffer from pointless compartementalization
> of resources such as in your proposal.
> 
No, this is not what I want.  :-)

> The crucial point is that if at one particular instance the
> i386.buildworld needs only one token, it will only claim
> one token, and the other jobs can get more tokens and start
> more things.  Once the i386 needs to run more things in
> parallel, it will compete for tokens again and run as much
> in parallel as it can get tokens for.
> 
I well understand this.

> In your scenario, the tokens not used by i386 cannot be
> used by the other buildworlds and they are wasted.
> 
Not wasted, they just don't share the same pool of jobs.  ;)

> >If you're by chance familiar with how you can ask Oracle to
> >optimize your queries, you can either optimize for ALL ROWS or
> >for FIRST ROWS, and it's up to you to decide on this.  The
> >"rows" in "make universe" are results of doing buildworld
> >and buildkernel, and you have control over this.
> 
> This is not a database.
> 
> This is not even a cartesian problem space.
> 
I'm interested in completing i386 buildworld as fast as
possible.  In "make -j12 universe", the i386 world will
be built in approximately (TOTAL) seconds (let's assume
for the moment that we don't build kernels).  If I use
"make universe JFLAG=-j12", i386 world will complete in
(TOTAL / 6) seconds.  If I use "make -j2 universe
JFLAG=-j6", i386 and pc98 worlds will complete in
(TOTAL / 6 * 2) seconds.  Can you see a difference?
I'm interested in "first rows".  And I'm not saying
I'm always interested in them, just want to be able
to test my patches as quickly as possible.  ;)

> >Do you disagree with anything of the above?
> 
> Yes, I disagree with pretty much all of it.  It is clear that
> you have not actually tried running with the code which is in
> the tree.
> 
Why?  It's clear that I understand how it works, and
you fail to see _my_ point.  ;)

> >Can you agree to the below patch?  It doesn't change anything
> >except when JFLAG is set explicitly to request another level
> >of parallelism for building individual worlds and kernels?
> 
> The patch is pointless, but harmless once people learn not to set
> JFLAG.
> 
> But I will not agree to this patch _until_ you show me _actual_ _real
> _life_ data that shows it performs better than just running without
> a JFLAG at all.
> 
You disagree that the i386 part of "make universe" will
complete faster in my scenario, while still providing an
overall limit of 12 jobs?  I understand the drawback of
my approach -- that the total time for "make universe"
will be longer in my case.

I now feel like everything that should have been said
was said, and this thread should die.  ;)


Cheers,
-- 
Ruslan Ermilov
ru_at_FreeBSD.org
FreeBSD committer