Re: Some performance measurements on the FreeBSD network stack

On Tue, Apr 24, 2012 at 02:16:18PM +0000, Li, Qing wrote:
> >
> >From previous tests, the difference between flowtable and
> >routing table was small with a single process (about 5% or 50ns
> >in the total packet processing time, if i remember well),
> >but there was a large gain with multiple concurrent processes.
> >
> 
> Yes, that sounds about right when we did the tests a long while ago.
> 
> >
> > Removing flowtable increases the cost in ip_output()
> > (obviously) but also in ether_output() (because the
> > route does not have a lle entry so you need to call
> > arpresolve on each packet). 
> >
> 
> Yup.
> 
> >
> > So in revising the route lookup i believe it would be good
> > if we could also get at once most of the info that
> > ether_output() is computing again and again.
> >
> 
> Well, the routing table no longer maintains any lle info, so there
> isn't much to copy out the rtentry at the completion of route
> lookup.
> 
> If I understood you correctly, you do believe there is a lot of value
> in Flowtable caching concept, but you are not suggesting we reverting
> back to having the routing table maintain L2 entries, are you ?

I see a lot of value in caching in general.

Especially for a bound socket it seems pointless to lookup the
route, iface and mac address(es) on every single packet instead of
caching them. And, routes and MAC addresses are volatile anyways
so making sure that we do the lookup 1us closer to the actual use
gives no additional guarantee.

The frequency with which these info (routes and MAC addresses)
change clearly influences the mechanism to validate the cache.
I suppose we have the following options:

- direct notification: a failure in a direct chain of calls
  can be used to invalidate the info cached in the socket.
  Similarly, some incoming traffic (e.g. TCP RST, FIN,
  ICMP messages) that reach a socket can invalidate the cached values
- assume a minimum lifetime for the info (i think this is what
  happens in the flowtable) and flush it unconditionally
  every such interval (say 10ms).
- if some info changes infrequently (e.g. MAC addresses) one could
  put a version number in the cached value and use it to validate
  the cache.

cheers
luigi