Re: netmap: extension to store user data per packet/slot?

On Wed, Nov 12, 2014 at 11:16 AM, Slawa Olhovchenkov <slw_at_zxy.spb.ru> wrote:

> On Tue, Nov 11, 2014 at 10:13:54PM +0100, Franco Fichtner wrote:
>
> > Hi Luigi,
> > hi all,
> >
> > so I was running into logistics issues with netmap(4)
> > with regard to zero-copy and redirection through pipes:
> > working on a load-balancing framework revealed that it
> > is very hard to track a packet's origins to later move
> > it onward to the respective outgoing interface, be it
> > another device or the host stack.
> >
> > Long story short: user data needs to be stored for the
> > packet buffer or slot.
>
> I think need configurable (by sysctl) space recerved before packet.
> This is may be used as user data. Or for insert VLAN/MPLS/QinQ/etc
> headers.
>
>
​this is yet another requirement: not just metadata but
also encapsulation.

For the records, the VALE switch does have TSO support (implemented
through the VHOST header) so that VMs can pass large segments
across a switch and they are properly split when traffic goes
to a physical interface or a port that does not support the
header. We also support scatter-gather I/O at least on the switch
(haven't implemented this feature yet on NICs).

But please consider that following this route we end up more or less
into the same complications that afflict the standard stack:
everything is configurable and decided at runtime, and the code
becomes a maze of conditionals or indirect function calls
with little chance of optimisations.

Also, it's not that one sysctl works for all cases. Different ports
typically have different encapsulation sizes, NICs may have
alignment constraints (even those who don't suffer if buffers
not 64-byte aligned), so you'll end up with scatter-gather I/O
or copying anyways.

After two years of experience with netmap i am not so sure
anymore that zero copy makes much sense, except perhaps for
the case of large packets (but i am not so sure about that, either).

Apart from benchmarks, if you want to do something useful with the
packets you need to read the header, at which point the concerns
on having data in cache or not are less significant and the cost
of the copy is heavily reduced. Tracking ownership of buffers
(which is needed for zero copy) is also expensive even when
they are not shared (and we have great trouble in managing the
"extra buffers" we recently added to the netmap API to support
zero-copy, to the point that I am tempted to remove the feature.

cheers
luigi