Re: Coalescing pipe allocation

On Sun, 1 Feb 2004, Dag-Erling Smørgrav wrote:

> Robert Watson <rwatson_at_FreeBSD.org> writes:
> > I have not yet macro-benchmarked this change.  It will either have a
> > slight performance benefit if half-closed pipes aren't used extensively or
> > it doesn't make a difference, be a wash, or if half-closed pipes turn up a
> > lot, potentially slightly negatively impact performance (seems unlikely).
> 
> [you may want to check the -current archives for silby_at_'s excellent
> message about pipe memory usage last Tuesday]

Yeah, I just chatted with Silby and he sent me a copy of the message.  His
observations all look pretty on-target to me, and suggested strategies
look good.

> I think our biggest problem with pipes is not half-open pipes but rather
> their absence; most of the code that uses pipes only uses one direction
> and does not bother to shut the other down.  This is certainly the case
> for sh(1) and make(1), which I suspect are the heaviest pipe consumers
> in a typical system (especially in one running our favorite benchmark
> tool, 'make buildworld'.)  The result is that we use almost twice as
> much memory for pipes as we need to.
> 
>  * solution: allocate pipe buffers on first-use rather than at pipe
>    creation time.

Yeah, I was interested by this observation -- many applications don't even
think pipes are bi-directional, meaning that doing work allocating pipe
map space for both directions basically wastes half the work.  This change
should be pretty easy, given that we already have code in to do a resize
on write.  It's probably as simply dropping the call to pipe_create() in
my version of the code, and perhaps making sure that pipespace() checks
that case.

FYI, the vast majority of the cost of pipe creation appears to be in
setting up those mappings.  I ran some mutex profiling and the amount of
work getting done is pretty high -- we should be able to reap a lot of
benefit by doing even a little less work there.  Doing half the work would
do wonders.  Caching sets of mappings for reuse would also help a lot, I
suspect, and we may be able to use UMA to help manage that (not sure yet
how well that will work). 

> Another problem with pipes is that the default pipe buffer size is too
> large. 
> 
>  * solution: reduce the initial allocation to a single page, and grow
>    the buffer later if necessary (up to a limit og four or eight
>    pages)
> 
> One problem we can't solve easily is the fact that pipe buffers are
> allocated directly from the pipe vm map, which is a very inefficient
> operation in FreeBSD.  I have a tentative design in my notebook for a
> more efficient data structure for vm maps, allowing vm_map_find() to run
> in logarithmic rather than linear time, but this hasn't been high on my
> priority list as it's fairly hard to get right. 
> 
> BTW, as I am currently working on some of these issues, I'd be happy if
> you could commit your patches fairly quickly so I don't have to start
> all over again once they hit the tree. 

Ok, sounds good.  I got Juli to do a technical review, and have been
running a variety of tests.  I should probably just commit it so it's
there and then we can refine.  I would imagine we can get another 30% of
the cost out of pipe creation (at least) through a bit of refinement of
just the sort we're discussing. 

Look for a commit sometime in the next hour or two -- I've got a couple of
other people running big builds with it, etc, to make sure nothing shakes
loose. 

Robert N M Watson             FreeBSD Core Team, TrustedBSD Projects
robert_at_fledge.watson.org      Senior Research Scientist, McAfee Research