802.1Q vlan performance.

Hi

While doing some experimentation and work on ipfw to see where I
could improve performance for our virtualised firewall I came across
the following comment in sys/net/if_vlan.c:

 * The VLAN_ARRAY substitutes the dynamic hash with a static array
 * with 4096 entries. In theory this can give a boots(sic) in processing,
 * however on practice it does not. Probably this is because array
 * is too big to fit into CPU cache.

Being curious and having determined the main throughput bottleneck
to be the vlan driver, I thought that I'd test the assertion.  I
have have 506 vlans on this machine.

With VLAN_ARRAY unset, ipfw disabled, fastforwarding enabled,
vlanhwtag enabled on the interface, the fastest forwarding rate I
could get was 278kpps (This was a steady decrease from 440kpps with
24 vlans linearly proportional to the number of vlans).

With exactly the same configuration, but the vlan driver compiled
with VLAN_ARRAY defined, the forwarding rate of the system is back
at 440kpps.

The testbed looks like this:

|pkt gen |            | router |                    | pkt rec  |
| host   |vlan2 vlan2 |        |vlan1002   vlan1002 |  host    |
|netperf |----------->|        |------------------->| netserver|
|        |em0     em0 |        |em1             em0 |          |

The router has vlan2 to vlan264 and vlan1002 through vlan1264 in
22 blocks of 23 vlan groups (a consequence of 24 port switches to
to tag/untag for customers).  The pkt gen and recieve host both
have 253 vlans.

Can anyone suggest a good reason not to turn this option on by
default.  It looks to me like it dramatically improves performance.

Ian

--
Ian Freislich