Re: Anomalous performance increase from mutex profiling

On Mon, Apr 17, 2006 at 01:14:40AM -0400, Kris Kennaway wrote:

> * Our best guess is that mutex profiling is doing something that
> reduces contention on this very heavily contended mutex (unp), but I'd
> like to know what is happening precisely so I can maybe make use of
> it.
> 
> Can anyone think of what may be happening that I've missed?

I think it is just doing effectively the same thing as my exponential
spin backoff patch, namely introducing delays with the effect of
reducing common memory accesses.  When I turn the maximum spin backoff
limit *way* up (from 1600 to 51200) I get performance that slightly
exceeds what I see from mutex profiling alone (adding mutex profiling
again on top of this gives a small further increase, but only a few %
and so probably achievable by further increasing the backoff limit).

A limit of 51200 is not an appropriate default since it penalizes the
common case of light to moderate contention.  The point is that here
basically all 12 CPUs are spinning on a single lock
(kern/uipc_usrreq.c:308), so it's massively over-contended and all we
can do is mitigate the effects of this.

On this system, the maximum supersmack performance (3700 queries/sec)
comes when there are only 6 clients, so (as jasone eloquently put it)
with 10 clients the difference between 2300 queries/sec (with absurdly
high backoff limits or mutex profiling) and 1450/sec (with reasonable
backoff limits) is the difference between "slow" and "ass slow".

Fortunately rwatson is working on breaking up this lock, so that
should help a lot here :-)

Kris