Re: Some performance measurements on the FreeBSD network stack

On Thu, Apr 19, 2012 at 11:20:00PM +0200, Andre Oppermann wrote:
> On 19.04.2012 22:46, Luigi Rizzo wrote:
...
> >What might be moderately expensive are the critical_enter()/critical_exit()
> >calls around individual allocations.
> 
> Can't get away from those as a thread must not migrate away
> when manipulating the per-CPU mbuf pool.

i understand.

> >The allocation happens while the code has already an exclusive
> >lock on so->snd_buf so a pool of fresh buffers could be attached
> >there.
> 
> Ah, there it is not necessary to hold the snd_buf lock while
> doing the allocate+copyin.  With soreceive_stream() (which is

it is not held in the tx path either -- but there is a short section
before m_uiotombuf() which does

	...
	SOCKBUF_LOCK(&so->so_snd);
	// check for pending errors, sbspace, so_state
	SOCKBUF_UNLOCK(&so->so_snd);
	...

(some of this is slightly dubious, but that's another story)

> >But the other consideration is that one could defer the mbuf allocation
> >to a later time when the packet is actually built (or anyways
> >right before the thread returns).
> >What i envision (and this would fit nicely with netmap) is the following:
> >- have a (possibly readonly) template for the headers (MAC+IP+UDP)
> >   attached to the socket, built on demand, and cached and managed
> >   with similar invalidation rules as used by fastforward;
> 
> That would require to cross-pointer the rtentry and whatnot again.

i was planning to keep a copy, not a reference. If the copy becomes
temporarily stale, no big deal, as long as you can detect it reasonably
quiclky -- routes are not guaranteed to be correct, anyways.

> >- possibly extend the pru_send interface so one can pass down the uio
> >   instead of the mbuf;
> >- make an opportunistic buffer allocation in some place downstream,
> >   where the code already has an x-lock on some resource (could be
> >   the snd_buf, the interface, ...) so the allocation comes for free.
> 
> ETOOCOMPLEXOVERTIME.

maybe. But i want to investigate this.

cheers
luigi