Re: Coalescing pipe allocation

On Sun, 1 Feb 2004, Dag-Erling Smørgrav wrote:

> Robert Watson <rwatson_at_FreeBSD.org> writes:
> > FYI, the vast majority of the cost of pipe creation appears to be in
> > setting up those mappings.  I ran some mutex profiling and the amount of
> > work getting done is pretty high -- we should be able to reap a lot of
> > benefit by doing even a little less work there.
> 
> Like I said, setting up a VM mapping is very inefficient in FreeBSD. 
> This, and the fd allocation issue which I fixed earlier this year, were
> the two areas where fefe's scalability benchmark really trounced us last
> October. 

FYI, on that topic, I ran into the following WITNESS warning today while
pounding on the pipe subsystem:

login: lock order reversal
 1st 0xc6585138 filedesc structure (filedesc structure) _at_
kern/kern_descrip.c:11
75
 2nd 0xc0937960 Giant (Giant) _at_ vm/uma_core.c:2048
Stack backtrace:
backtrace(c087d47f,c0937960,c08794df,c08794df,c0892fb4) at backtrace+0x17
witness_checkorder(c0937960,9,c0892fb4,800,c066e432) at
witness_checkorder+0x6f8
_mtx_lock_flags(c0937960,0,c0892fab,800,c08d0e00) at _mtx_lock_flags+0x9d
uma_large_free(c64ed7e0,c063890a,c6585138,8,c087714b) at
uma_large_free+0xae
free(c66d9000,c08d0e00,400,497,c6585300) at free+0x11e
fdgrowtable(c6585100,800,400,4be,2dc0) at fdgrowtable+0x1b4
fdalloc(c6487bd0,0,e5bd5c9c,52a,c658fa50) at fdalloc+0x93
falloc(c6487bd0,e5bd5ccc,e5bd5cd4,164,0) at falloc+0x1f3
pipe(c6487bd0,e5bd5d14,c0898495,43a,0) at pipe+0x151
syscall(2f,2f,2f,bfbfec50,bfbfec58) at syscall+0x2a0
Xint0x80_syscall() at Xint0x80_syscall+0x1d
--- syscall (42), eip = 0x28097e3f, esp = 0xbfbfec0c, ebp = 0xbfbfec28 ---
kern.ipc.maxpipekva exceeded; see tuning(7)

The test scenario consisted of forcing paging while running lots of pipe
allocation code in parallel on SMP:

robert_at_none:~> more page.c
#include <stdio.h>
#include <stdlib.h>

int
main(int argc, char *argv[])
{
        char *p;

        while (1) {
                p = malloc(getpagesize());
                if (p == NULL) {
                        perror("malloc");
                        while (1);
                }
                p[0] = '\0';
        }

}

robert_at_none:~> more test.c 
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>

int
main(int argc, char *argv[])
{
        int fd[2];

        while (1) {
                if (pipe(fd) == -1) {
                        perror("pipe");
                        exit(-1);
                }
        }
}

I'm not quite sure when it transpired, but somewhere during the following:


% ./page & ./page & ./page & ./page &
% while (1)
    ./test & ./test & ./test &
  end

So there was probably pretty strong Giant contention while the paging
processes were initially allocation memory.  I was somewhat surprised to
see UMA hit Giant there. 

Robert N M Watson             FreeBSD Core Team, TrustedBSD Projects
robert_at_fledge.watson.org      Senior Research Scientist, McAfee Research