Re: vmstat's entries type

On Fri, 2006-Jul-28 14:47:01 +0100, Brian Candler wrote:
>On Fri, Jul 28, 2006 at 09:28:36AM -0400, John Baldwin wrote:
>> 	lock incl counter
>> 	jnc 1f
>> 	lock incl counter+4
>> 1:

This approach still requires the reader to loop with something like
	do {
		a.lo = counter.lo;
		a.hi = counter.hi;
		b.lo = counter.lo;
		b.hi = counter.hi;
	} while (a.hi != b.hi || a.lo > b.lo);
to ensure that the reader doesn't read the middle of an update.

>The 'polling' argument says just do
>    lock incl counter
>and poll all counters every 5 minutes, looking for a wrap. I think that's
>almost certainly going to be cheaper, as long as you can keep track of where
>all these counters are located.

lock prefixes are always going to be extremely expensive on a MP
system because they require physical bus cycles.  RISC architectures
usually only have TAS lock primitives (because "inc mem" doesn't
exist) and so require a spinlock to perform an atomic update.

In a MP configuration where it doesn't particularly matter if a
particular update gets counted this time or next time, I think the
cheapest option is to have per-CPU 32-bit counters (so no locks are
needed to update the counters) with a polling function to accumulate
all the individual counters into a 64-bit total.  This pushes the cost
from the update (very frequent) into the read (which is relatively
infrequent), for a lower overall cost.

This turns the update into something like:
	PCPU_SET(counter, PCPU_GET(counter)+1);
or
	incl	%fs:counter
(no locks or atomic operations)

Whilst the poll/read pseudo code looks something like
	lock counter
	foreach cpu {
		uint32 a = cpu->counter;
		uint32 b = cpu->last_counter;
		uint32 c = counter.lo;
		if (b > a)
			counter.hi++;
		counter.lo += a - b;
		if (counter.lo < c)
			counter.hi++;
		cpu->last_counter = a;
	}
	unlock counter;
(the lock prevents multiple readers updating counter simultaneously).

You execute this whenever a reader wants the counter value (eg via
SYSCTL_PROC), as well as a rate sufficient to prevent missing wraps
(eg every 2 seconds for a 10g byte counter).  This rate is sufficiently
lower than the update rate to make the whole exercise worthwhile.

-- 
Peter Jeremy