Re: [TEST/REVIEW] CPU accounting patches

On Fri, Jan 27, 2006 at 01:48:31PM -0700, Scott Long wrote:
> Garance A Drosehn wrote:
> 
> >At 11:58 AM +0100 1/25/06, Poul-Henning Kamp wrote:
> >
> >>
> >>With my definition you would be more likely to see lower numbers
> >>maybe
> >>    user 0.20  sys 0.03 real 4.00
> >>
> >>And they would have meaning, they should be pretty much the same
> >>no matter what speed your CPU runs at any instant in time.
> >>
> >>In theory, it should be possible to compare user/sys numbers
> >>you collect while running at 75 MHz with the ones you got
> >>under full steam at 1600 MHz.
> >>
> >>In practice however, things that run on the real time, HZ
> >>interrupting to run hardclock() for instance, will still make
> >>comparison of such numbers quite shaky.
> >>
> >>But at least they will not be random as they are now.
> >
> >
> >Here at RPI we used to have a mainframe, and we used to charge
> >by the CPU second, so I am familiar with that side of the
> >question.  However, I am not too concerned by it for my own
> >interests.  For one, we don't charge by CPU second any more.  For
> >two, even if we did start charging again, we would just come up
> >with some other metric, or simply pick a different rate for
> >charging.
> >
> >The other big usage for timing programs is to compare the
> >performance of various algorithms.  We have always had users who
> >cared very much about the accuracy and consistency of such
> >measurements, whether or not we were charging people by the "CPU
> >second".  Based on the above description, the new CPU accounting
> >patches will make those comparisons more meaningful, since the
> >values measured will be the same no matter what speed the CPU is
> >running at.  As such, I think it's a good idea, even if we ignore
> >the performance improvement.
> >
> >Rambling part:
> >
> >Getting back to the question of charging, I can almost convince
> >myself that these changes are also a good idea for when those
> >values are used for charging.  When we (RPI) charged for CPU time,
> >we weren't really charging "for CPU seconds".  We were charging
> >to say "when we are forced to buy a new computer because this
> >computer is maxed out, then how much of that load (and thus the
> >expense for the new computer) is the fault of any given user?".
> >
> >Thus, if we had a computer which could vary it's speed, we don't
> >really care about "running out of CPU seconds" if the CPU is in
> >fact running at half-speed.  We only incur the cost of a new
> >machine once we "run out of CPU seconds" when it is running at
> >*maximum* speed.
> >
> >Furthermore, if we had a load which was low enough that we
> >*could* get it all done by running the CPU at half-speed, then
> >we (as the computer center) would *prefer* to run it at half-
> >speed.  That way, we reduce power and cooling costs.  However,
> >we will create extreme hostility in our users if we save that
> >money, only to charge them twice as much because they are now
> >forced to use "twice as many CPU seconds" when we run the CPU
> >at half-speed.  Oddly enough, consistency is also the big issue
> >when it comes to charging.  The user expects to see the exact
> >same charge every time they run a specific job, and not see
> >their charges vary by a dramatic amount due issues they have
> >no control over.
> >
> >Poul-Henning says the values will "not be as random as they are now".
> >If someone *is* charging by the CPU second, then they don't want
> >those values to be "random".  People who receive random bills can
> >get really really hostile, perhaps to the point of bringing in
> >lawyers.  (and I have seen that happen).
> >
> >So, it seems to me that this change is *always* the behavior that
> >everyone would prefer.  Yes, we have to describe it.  And maybe we
> >should call the value something other than "CPU seconds" to make
> >that clear, although I don't know what would be a better name.  But
> >I think I have convinced myself that there is no downside to these
> >proposed changes.
> >
> >...Assuming the changes work, of course!
> >
> 
> Just call it 'cpu cycles'.  If I have a job that calls nop 1 billion 
> times, then I expect to get charged for 1 billion cycles regardless of
> if it takes 1 second or 5 days to run.  I agree completely with your
> argument for consistency and that this will improve consistency and 
> predictability.

I agree as well.  Certainly if we were charging for use of our cluster,
this is what we'd want.  While I probably wouldn't run powerd on the
cluster, I and thinking about seeing if I can step down the CPU speed
when there aren't any queued jobs on the machine.  That could save
significant power some of the time (I'm in the process of upgrading the
cluster portion of our server room to install 300KVA (~KW) of power and
plan to use it all within a year or two).

Once we have the infrastructure to deal with this correctly, an
intresting test for someone to run would be to look at disk and memory
bound applications at different CPU speeds.  I suspect you'd find that
while wallclock increased at lower CPU speeds, cpu cycles would decrease
for many workloads because the relative bandwidth of storage and maybe
memory would increase.

-- Brooks

-- 
Any statement of the form "X is the one, true Y" is FALSE.
PGP fingerprint 655D 519C 26A7 82E7 2529  9BF0 5D8E 8BE9 F238 1AD4