Re: serious networking (em) performance (ggate and NFS) problem

    Polling should not produce any improvement over interrupts for EM0.
    The EM0 card will aggregate 8-14+ packets per interrupt, or more.
    which is only around 8000 interrupts/sec.  I've got a ton of these 
    cards installed.

    # mount_nfs -a 4 dhcp61:/home /mnt
    # dd if=/mnt/x of=/dev/null bs=32k
    # netstat -in 1
		input        (Total)           output
       packets  errs      bytes    packets  errs      bytes colls
	 66401     0   93668746       5534     0     962920     0
	 66426     0   94230092       5537     0    1007108     0
	 66424     0   93699848       5536     0     963268     0
	 66422     0   94222372       5536     0    1007290     0
	 66391     0   93654846       5534     0     962746     0
	 66375     0   94154432       5532     0    1006404     0

                                                          zfod   Interrupts
Proc:r  p  d  s  w    Csw  Trp  Sys  Int  Sof  Flt        cow    8100 total
     1        9      6211    7   75 8100    4   12  88864 wire   7873 mux irq10
                                                    10404 act         ata0 irq14
19.2%Sys   0.0%Intr  0.0%User  0.0%Nice 80.8%Idl   864476 inact       ata1 irq15
|    |    |    |    |    |    |    |    |    |      58152 cache       mux irq11
==========                                           2992 free    227 clk irq0


    Note that the interrupt rate is only 7873 interrupts per second
    while I am transfering 94 MBytes/sec over NFS (UDP) and receiving
    over 66000 packets per second (~8 packets per interrupt).

    If I use a TCP mount I get just about the same thing:

    # mount_nfs -T -a 4 dhcp61:/home /mnt
    # dd if=/mnt/x of=/dev/null bs=32k
    # netstat -in 1

            input        (Total)           output
   packets  errs      bytes    packets  errs      bytes colls
     61752     0   93978800       8091     0     968618     0
     61780     0   93530484       8098     0     904370     0
     61710     0   93917880       8093     0     968128     0
     61754     0   93491260       8095     0     903940     0
     61756     0   93986320       8097     0     968336     0


Proc:r  p  d  s  w    Csw  Trp  Sys  Int  Sof  Flt        cow    8145 total
           5  8     22828   13 5490 8146   13   11 141556 wire   7917 mux irq10
                                                     7800 act         ata0 irq14
26.4%Sys   0.0%Intr  0.0%User  0.0%Nice 73.6%Idl   244872 inact       ata1 irq15
|    |    |    |    |    |    |    |    |    |          8 cache       mux irq11
=============                                      630780 free    228 clk irq0

    In this case around 8000 interrupts per second with 61700 packet per
    second incoming on the interface (around ~8 packets per interrupt).
    The extra interrupts are due to the additional outgoing TCP ack traffic.

    If I look at the systat -vm 1 output on the NFS server it also sees
    only around 8000 interrupts per second, which isn't saying much other
    then it's transmit path (61700 pps outoging) is not creating an undue
    interrupt burden relative to the receive path.

						-Matt