Re: Some performance measurements on the FreeBSD network stack

Comments inline below:

On Fri, Apr 20, 2012 at 4:44 PM, Luigi Rizzo <rizzo_at_iet.unipi.it> wrote:
> On Thu, Apr 19, 2012 at 11:06:38PM +0200, K. Macy wrote:
>> On Thu, Apr 19, 2012 at 11:22 PM, Luigi Rizzo <rizzo_at_iet.unipi.it> wrote:
>> > On Thu, Apr 19, 2012 at 10:34:45PM +0200, K. Macy wrote:
>> >> >> This is indeed a big problem. ?I'm working (rough edges remain) on
>> >> >> changing the routing table locking to an rmlock (read-mostly) which
>> >> >
>> >>
>> >> This only helps if your flows aren't hitting the same rtentry.
>> >> Otherwise you still convoy on the lock for the rtentry itself to
>> >> increment and decrement the rtentry's reference count.
>> >>
>> >> > i was wondering, is there a way (and/or any advantage) to use the
>> >> > fastforward code to look up the route for locally sourced packets ?
>> >
>> > actually, now that i look at the code, both ip_output() and
>> > the ip_fastforward code use the same in_rtalloc_ign(...)
>> >
>> >> >
>> >>
>> >> If the number of peers is bounded then you can use the flowtable. Max
>> >> PPS is much higher bypassing routing lookup. However, it doesn't scale
>> >> to arbitrary flow numbers.
>> >
>> > re. flowtable, could you point me to what i should do instead of
>> > calling in_rtalloc_ign() ?
>>
>> If you build with it in your kernel config and enable the sysctl
>> ip_output will automatically use it for TCP and UDP connections. If
>> you're doing forwarding you'll need to patch the forwarding path.
>
> cool.
> For the records, with "netsend 10.0.0.2 ports 18 0 5" on an ixgbe
> talking to a remote host i get the following results (with a single
> port netsend does a connect() and then send(), otherwise it
> loops around a sendto() )
>

Sorry, 5000 vs 5000-5001 means 1 vs 2 streams? Does this mean for a
single socket the overhead is less without it compiled in than with it
compiled in but enabled? That is certainly different from what I see
with TCP where I see a 30% increase in aggregate throughput the last
time I tried this (on IPoIB).

For the record the M_FLOWID is used to pick the transmit queue so with
multiple streams you're best of setting it if your device has more
than one hardware device queue.

>        net.flowtable.enabled   port            ns/pkt
>        -----------------------------------------------------
>        not compiled in         5000             944    M_FLOWID not set
>        0 (disable)             5000            1004
>        1 (enable)              5000             980
>
>        not compiled in         5000-5001       3400    M_FLOWID not set
>        0 (disable)             5000-5001       1418
>        1 (enable)              5000-5001       1230
>
> The small penalty when flowtable is disabled but compiled in is
> probably because the net.flowtable.enable flag is checked
> a bit deep in the code.
>
> The advantage with non-connect()ed sockets is huge. I don't
> quite understand why disabling the flowtable still helps there.

Do you mean having it compiled in but disabled still helps
performance? Yes, that is extremely strange.

-Kip


-- 
   “The real damage is done by those millions who want to 'get by.'
The ordinary men who just want to be left in peace. Those who don’t
want their little lives disturbed by anything bigger than themselves.
Those with no sides and no causes. Those who won’t take measure of
their own strength, for fear of antagonizing their own weakness. Those
who don’t like to make waves—or enemies.

   Those for whom freedom, honour, truth, and principles are only
literature. Those who live small, love small, die small. It’s the
reductionist approach to life: if you keep it small, you’ll keep it
under control. If you don’t make any noise, the bogeyman won’t find
you.

   But it’s all an illusion, because they die too, those people who
roll up their spirits into tiny little balls so as to be safe. Safe?!
>From what? Life is always on the edge of death; narrow streets lead to
the same place as wide avenues, and a little candle burns itself out
just like a flaming torch does.

   I choose my own way to burn.”

   Sophie Scholl