Re: Flowtables -- any tuning hints?

Kip Macy wrote:
> On Sat, Jul 11, 2009 at 10:24 AM, Scott Ullrich<sullrich_at_gmail.com> wrote:
>   
>> Hello Freebsd-current_at_ folks,
>>
>> I see with the commit "svn commit: r191259 - head/sys/netinet"
>> flowtables have been added..  Cool!
>>
>> Does anyone have any tuning hints for this addition -- specifically
>> how much memory does the hash table consume?   Or better yet does any
>> documentation exist for this newly added feature?
>>
>> Looking for an easy way to calculate max flows for the amount of
>> memory installed in a FreeBSD machine.
>>     
>
>
> You want to avoid hash collisions. So, generally speaking you want the
> hash table to be sized 2x larger than the number of unique connection
> destinations.  You want the maximum number of flows to be as large as
> the maximum number of unique destinations x number of cores. When you
> get to the case of hundreds of thousands of unique destinations as in
> the case of a small ISP doing IP forwarding, you're probably better
> off disabling the flowtable. For most other workloads its likely to be
> a clear win. Running a process on an 8-core system with 8 threads each
> calling sendto(...) with 10 bytes I can push 3.5 - 4Mpps (with cxgb -
> you won't get this with most cards) with the flowtable enabled. With
> the flowtable disabled lock contention causes performance to degrade
> to 330kpps with the aforementioned workload.
>   
This is interesting functionality, but I think we need to look at it a 
bit closer for our use case. Is there any benefit in running this in a 
firewall scenario? That's primarily what Scott and I (pfsense) are 
interested in. In our world, if you're pushing 50Kpps+, you're almost 
certainly falling into the "small ISP doing IP forwarding" scenario with 
hundreds of thousands of unique destinations. Where we usually see these 
kinds of loads are small ISPs, web hosting companies, or universities 
(which are functionally not much diff from a small ISP), all of which 
I'm familiar with falling into the "better off disabling" category. I 
also suspect pf's locking negates some or all of the benefits here.

I suspect it's not applicable to the specific workload our users 
normally have, where you're almost entirely doing IP forwarding, and 
initiating very little if any traffic. bz_at_ said it's not something you 
want on a router. Is that a fair assessment? 

best,
Chris