Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets)

On Sun, May 07, 2006 at 05:04:26PM -0400, Kris Kennaway wrote:

> >    477     23472709     2810986     8      5671248      1900047 kern/kern_synch.c:220 (process lock)
> > 
> > The top 10 heavily contended mutexes are very different (but note the
> > number of mutex acquisitions, column 3, is about the same).
> > 
> > There is not much contention on unp_global_mtx any longer, but there
> > is a lot more on some of the other mutexes, especially the process
> > lock via msleep().  Off-hand I don't know what is the cause of this
> > bottleneck (note: libthr is used as threading library and libpthread
> > is not ported to sparc64).

> i.e. the same number of msleep calls in both cases, but the patch
> shuffles them from sbwait onto everything else.

Oops, they're the same number because that was the size of my ktr
buffer, duh :) Nevertheless it's a sample from the end of the same
workload, so the distribution should be comparable.

> sigwait    +27%

This appears to be the only relevant msleep wait channel that is using
the process lock.

static int
kern_sigtimedwait(struct thread *td, sigset_t waitset, ksiginfo_t *ksi,
        struct timespec *timeout)
{
...
        td->td_sigmask = savedmask;
        SIGSETNAND(td->td_sigmask, waitset);
        signotify(td);
        error = msleep(&ps, &p->p_mtx, PPAUSE|PCATCH, "sigwait", hz);

i.e. several threads in the process are all sleeping for 1 tick and
then doing a thundering herd on the same proc lock when they wake up.

To confirm this I added a KTR tracepoint before the msleep call.  It's
indeed being called from the various threads in mysqld.  It's hard to
say for sure how many CPUs are contending at once, but plotting a
histogram of the time distribution of the calls indicates that several
CPUs are frequently executing this code in close proximity.

Kris