Re: [RFC] Removal of M_FLOWID flag from m_flags [WAS: Add support for hardware transmit rate limiting queues]

On 11/27/14 18:33, Adrian Chadd wrote:
> On 27 November 2014 at 09:25, Hans Petter Selasky <hps_at_selasky.org> wrote:
>> On 11/27/14 18:20, Adrian Chadd wrote:
>>>
>>> On 27 November 2014 at 09:18, Adrian Chadd <adrian_at_freebsd.org> wrote:
>>>>
>>>> Erk - SCTP folk - are you using the mbuf flowid field for something
>>>> SCTP specific?
>>>
>>>
>>> erk - yes, you are.
>>>
>>> It seems we're going to run into "what exactly should flowid be used
>>> for" problems.
>>>
>>
>> Hi,
>>
>> If the flowid has special meaning inside the SCTP, please define an own
>> "rsstype" for it. As far as I could see, it is only used to spread the
>> traffic on the network adapter and on the CPU cores.
>
> I'm more worried about at what layers we're going to be using the
> flowid values and that various pieces may stomp over other pieces.
>
> With the RSS stuff enabled, the IPv4 (and soon IPv6) input paths will
> re-hash an input frame if it doesn't have an RSS hash; it'll then
> overwrite whatever flowid value is in the mbuf. This is so no matter
> what the NIC hands us or what de-encapsulation hands us, we will
> always put that flow on into the right RSS bucket and a consistent
> CPU. This should mostly alleviate any out-of-order issues seen with
> internet-facing machines where things handle fragments and such.

Hi,

I'm not changing anything in the receive direction regarding the flowid. 
It should be exactly the same as before, except M_HASHTYPE_NONE which 
now indicates that flowid is not set.

>
> Then for the output side of things, it'll have to do a software RSS
> hash on frames that don't have an RSS hash. Right now I assume in the
> TCP path that the inp has an RSS flow setup in the inp and that the
> TCP timers and other assorted stuff uses the inp details.
>
> For the UDP output side of things I currently always re-calculate the
> RSS hash and stamp the mbuf with it before we send it out, again so it
> ends up on the same output RSS bucket and thus CPU / NIC queue.
>
> If the inp ends up with a different flowid (eg a hardware ring flowid) then:
>
> * it won't match up on the receive side, so the whole RSS lock
> contention avoidance thing can't happen;
> * there's a known mapping for RSS bucket -> CPU IDs (since we're not
> doing RSS bucket -> CPU ID reassignment yet) - but not for other
> flowid types to CPU IDs.

Not necessarily. Would could make a standard, that the lower X-bits of 
the flowid, always indicate RX/TX ring pair and the CPU core, and then 
the upper 32-X bits are free to use for other purposes.

>
> In general I think the tidyup patch looks fine and I'll do some more
> RSS testing once it's committed. (But you did sneak in your new hash
> type in the diff :-)

I'll put the new hash type in a separate patch. It doesn't belong there 
- you're right.

>
> I think we then need to actually plan out how this stuff should hold
> together before we all rush into it or we'll end up with a mess of
> components that can't actually interact together. I don't want to have
> to explain to people that they can't use SCTP, RSS and hardware ring /
> flow assignment at the same time. It's just going to be painful in the
> long run.

Do you have code not committed which plan to use the flowid in new areas 
of the FreeBSD kernel? I would like to have a list of usages for the 
flowid field?

--HPS