Re: [RFC] Removal of M_FLOWID flag from m_flags [WAS: Add support for hardware transmit rate limiting queues]

From: Adrian Chadd <adrian_at_freebsd.org>
Date: Thu, 27 Nov 2014 09:33:58 -0800
On 27 November 2014 at 09:25, Hans Petter Selasky <hps_at_selasky.org> wrote:
> On 11/27/14 18:20, Adrian Chadd wrote:
>>
>> On 27 November 2014 at 09:18, Adrian Chadd <adrian_at_freebsd.org> wrote:
>>>
>>> Erk - SCTP folk - are you using the mbuf flowid field for something
>>> SCTP specific?
>>
>>
>> erk - yes, you are.
>>
>> It seems we're going to run into "what exactly should flowid be used
>> for" problems.
>>
>
> Hi,
>
> If the flowid has special meaning inside the SCTP, please define an own
> "rsstype" for it. As far as I could see, it is only used to spread the
> traffic on the network adapter and on the CPU cores.

I'm more worried about at what layers we're going to be using the
flowid values and that various pieces may stomp over other pieces.

With the RSS stuff enabled, the IPv4 (and soon IPv6) input paths will
re-hash an input frame if it doesn't have an RSS hash; it'll then
overwrite whatever flowid value is in the mbuf. This is so no matter
what the NIC hands us or what de-encapsulation hands us, we will
always put that flow on into the right RSS bucket and a consistent
CPU. This should mostly alleviate any out-of-order issues seen with
internet-facing machines where things handle fragments and such.

Then for the output side of things, it'll have to do a software RSS
hash on frames that don't have an RSS hash. Right now I assume in the
TCP path that the inp has an RSS flow setup in the inp and that the
TCP timers and other assorted stuff uses the inp details.

For the UDP output side of things I currently always re-calculate the
RSS hash and stamp the mbuf with it before we send it out, again so it
ends up on the same output RSS bucket and thus CPU / NIC queue.

If the inp ends up with a different flowid (eg a hardware ring flowid) then:

* it won't match up on the receive side, so the whole RSS lock
contention avoidance thing can't happen;
* there's a known mapping for RSS bucket -> CPU IDs (since we're not
doing RSS bucket -> CPU ID reassignment yet) - but not for other
flowid types to CPU IDs.

In general I think the tidyup patch looks fine and I'll do some more
RSS testing once it's committed. (But you did sneak in your new hash
type in the diff :-)

I think we then need to actually plan out how this stuff should hold
together before we all rush into it or we'll end up with a mess of
components that can't actually interact together. I don't want to have
to explain to people that they can't use SCTP, RSS and hardware ring /
flow assignment at the same time. It's just going to be painful in the
long run.



-adrian
Received on Thu Nov 27 2014 - 16:34:00 UTC

This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:40:54 UTC