Re: [RFC] Add support for hardware transmit rate limiting queues [WAS: Add support for changing the flow ID of TCP connections]

On Wed, Aug 20, 2014 at 7:41 AM, Luigi Rizzo <rizzo_at_iet.unipi.it> wrote:

> On Wed, Aug 20, 2014 at 3:29 PM, Hans Petter Selasky <hps_at_selasky.org>
> wrote:
>
> > Hi Luigi,
> >
> >
> > On 08/20/14 11:32, Luigi Rizzo wrote:
> >
> >> On Wed, Aug 20, 2014 at 9:34 AM, Hans Petter Selasky <hps_at_selasky.org>
> >> wrote:
> >>
> >>  Hi,
> >>>
> >>> A month has passed since the last e-mail on this topic, and in the
> >>> meanwhile some new patches have been created and tested:
> >>>
> >>> Basically the approach has been changed a little bit:
> >>>
> >>> - The creation of hardware transmit rings has been made independent of
> >>> the
> >>> TCP stack. This allows firewall applications to forward traffic into
> >>> hardware transmit rings aswell, and not only native TCP applications.
> >>> This
> >>> should be one more reason to get the feature into the kernel.
> >>> ​...
> >>>
> >> ​the patch seems to include only part of the generic code (ie no ioctls
> >> for manipulating the rates, no backend code). Do i miss something ?
> >>
> >
> > The IOCTLs for managing the rates are:
> >
> > SIOCARATECTL, SIOCSRATECTL, SIOCGRATECTL and SIOCDRATECTL
> >
> > And they go to the if_ioctl callback.​
>
>
> ​i really think these new 'advanced' features should go
> through some ethtool-like API, not more ioctls.
> We have a strong need to design and implement such
> an API also to have a uniform mechanism to manipulate
> rss, queues and other NIC features.
>
>
>

There is no ethtool equivalent yet, but exposing them through a sysctl is
definitely the place to start before putting it straight in to ifconfig.
The ifnet API is already a bit of a mess.



> ​...​
> >
> >
> >
> >> I have a few comments/concerns:
> >>
> >> + looks like flowid and txringid are overlapped in scope,
> >>    both will be used (in the backend) to select a specific
> >>    tx queue. I don't have a solution but would like to know
> >>    how do you plan to address this -- does one have priority
> >>    over the other, etc.
> >>
> >
> > Not 100% . In some cases the flowID is used differently than the
> txringid,
> > though it might be possible to join the two. Would need to investigate
> > current users of the flow ID.
>
>
> ​in some 10G drivers i have seen, at the driver
> level the flowid is used on the tx path to assign
> packets to a given ​tx queue, generally to improve
> cpu affinity. Of course some applications
> may want a true flow classifier so they do not
> have to re-do the classification multiple times.
> But then, we have a ton of different classifiers
> with the same need -- e.g. ipfw dynamic rules,
> dummynet pipe/queue id, divert ports...
> Pipes are stored in mtags, which are very expensive
> so i do see a point in embedding them in the mbufs,
> it's just that going this path there is no end
> to the list.
>
>
>
The purpose of the flowid was to enforce packet ordering on transmit while
being large enough to store a RSS hash, potentially allowing input
consumers to use it to semi-uniquely label (srcip, srcport, dstip, dstport)
tuples. It seems to that the txringid would be almost entirely redundant.
Why not just let users set the flowid?


> If we can merge the flowID and the txringid into one field, would it be
> > acceptable to add an IOCTL to read/write this value for all sockets?
>
>
That sounds reasonable - although I have not thought through all the
implications.


-K