Network throughput problems in RELENG_7

I'm seeing a problem where a much faster quad-core host running RELENG_7
serves many fewer netrate/http requests per second (175/sec) than an
old, busy, UP 6.0 host (828/sec). The problem seems to be related to
latency and connection setup, as it shows up dramatically over a link
with 50-60 ms latency. Can you help?

"Gandalf" is the problem RELENG_7 host connected via em0 on a 100baseTX
full-duplex switch port.
"Boromir" is a 6.0 host for reference, connected via bge0 to another
100baseTX full-duplex port on the same switch.
"Theoden" is a 6.2 host remote (50-60 ms away) from gandalf and boromir.

Running tools/netrate/http against tools/netrate/httpd on gandalf and
boromir, serving a document with 35 bytes of content, I get this
(details including tcpdumps following my sig):

                dest    dest
        source  boromir gandalf
        ------  ------- -------
        theoden 828     175
        boromir -       18092
        gandalf 11493   -

The problem occurs using ab as well, but is much less pronounced when -k
is used (enabling keepalive and presumably HTTP pipelining). This makes
me wonder if the problem might have something to do with connection
setups.

I've verified that the switch port and the interface are configured
correctly (100baseTX full-duplex). I've also tried toggling polling and
the following sysctls, with no significant change:

        net.inet.tcp.tso
        net.inet.tcp.newreno
        net.inet.tcp.sack.enable
        net.inet.tcp.inflight.enable
        net.inet.tcp.syncookies_only
        net.inet.tcp.syncookies

What else can I try?

Thanks for any help you can give.

Matt




theoden-gandalf
===============

theoden:~$ /usr/obj/usr/src/tools/tools/netrate/http/http -s 2
216.193.240.4
1280 /short.html
187 transfers/second
0 errors/second

tcpdump:
http://gandalf.vpop.net/~mreimer/theoden-gandalf-netrate.dump.gz

---

theoden:~$ /usr/obj/usr/src/tools/tools/netrate/http/http -n 100 -s 2
216.193.240.4 1280 /short.html
175 transfers/second
0 errors/second

tcpdump:
http://gandalf.vpop.net/~mreimer/theoden-gandalf-netrate-n100.dump.gz



theoden-boromir
===============

theoden:~$ /usr/obj/usr/src/tools/tools/netrate/http/http -s 2
216.193.240.2
1280 /short.html
272 transfers/second
0 errors/second

tcpdump:
http://boromir.vpop.net/~mreimer/theoden-boromir-netrate.dump.gz

---

theoden:~$ /usr/obj/usr/src/tools/tools/netrate/http/http -n 100 -s 2
216.193.240.2 1280 /short.html
828 transfers/second
0 errors/second

tcpdump:
http://boromir.vpop.net/~mreimer/theoden-boromir-netrate-n100.dump.gz


boromir-gandalf
===============

boromir: $ /usr/obj/usr/src/tools/tools/netrate/http/http -n 100 -s 2
216.193.240.4 1280 /short.html
18092 transfers/second
0 errors/second

gandalf-boromir
===============

gandalf: $ /usr/obj/usr/src/tools/tools/netrate/http/http -n 100 -s 2
216.193.240.2 1280 /short.html
11493 transfers/second
0 errors/second




boromir sysctl and ifconfig:

http://boromir.vpop.net/~mreimer/boromir.sysctl.gz


# ifconfig bge0
bge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        options=1a<TXCSUM,VLAN_MTU,VLAN_HWTAGGING>
        inet6 fe80::2e0:81ff:fe60:3925%bge0 prefixlen 64 scopeid 0x1 
        inet 216.193.240.33 netmask 0xffffffff broadcast 216.193.240.33
        inet 216.193.240.34 netmask 0xffffffff broadcast 216.193.240.34
        inet 216.193.240.35 netmask 0xffffffff broadcast 216.193.240.35
        inet 216.193.240.36 netmask 0xffffffff broadcast 216.193.240.36
        inet 216.193.240.37 netmask 0xffffffff broadcast 216.193.240.37
        inet 216.193.240.17 netmask 0xffffffff broadcast 216.193.240.17
        inet 216.193.240.38 netmask 0xffffffff broadcast 216.193.240.38
        inet 216.193.240.39 netmask 0xffffffff broadcast 216.193.240.39
        inet 216.193.240.2 netmask 0xfffff800 broadcast 216.193.247.255
        inet 216.193.240.40 netmask 0xffffffff broadcast 216.193.240.40
        ether 00:e0:81:60:39:25
        media: Ethernet autoselect (100baseTX <full-duplex>)
        status: active

gandalf sysctl and ifconfig:
http://gandalf.vpop.net/~mreimer/gandalf.sysctl.gz

# ifconfig em0
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu
1500
        options=1cb<RXCSUM,TXCSUM,VLAN_MTU,POLLING,VLAN_HWCSUM,TSO4>
        ether 00:30:48:33:96:a4
        inet 216.193.240.4 netmask 0xfffff800 broadcast 216.193.247.255
        inet 216.193.240.65 netmask 0xffffffff broadcast 216.193.240.65
        inet 216.193.240.66 netmask 0xffffffff broadcast 216.193.240.66
        inet 216.193.240.67 netmask 0xffffffff broadcast 216.193.240.67
        inet 216.193.240.68 netmask 0xffffffff broadcast 216.193.240.68
        media: Ethernet 100baseTX <full-duplex>
        status: active