mbuf leak with SMP and debug.mpsafenet=1

From: Andrew Gallatin <gallatin_at_cs.duke.edu>
Date: Thu, 14 Oct 2004 09:53:48 -0400 (EDT)
When bringing up our out-of-tree network driver, I noticed a pretty
bad mbuf leak.  Unfortunately, I think that the leak is in the system,
and not in our driver because mbufs continue to leak at a rate of a
few hundred per minute even when using the fxp interface.  The leak is
present after a reboot w/o loading our driver.

You can duplicate this easily with netperf/netserver.
Assume "scream" is the freebsd SMP host.  Start netserver there.
"scream" is a 3.06GHz P4 running RELENG_5 updated this morning.
Its a HTT P4, running an SMP kernel.  No witness/invariants.  No
zero-copy sockets.

Log into another host and do:

% while 1
while?  ssh  'netstat -m | grep mbufs'
while?  netperf224 -Hscream -tUDP_STREAM -l60 -- -m 1 >& /dev/null
while? end

441 mbufs in use
761 mbufs in use
1195 mbufs in use
1740 mbufs in use
2205 mbufs in use
2753 mbufs in use
3131 mbufs in use
3604 mbufs in use
4042 mbufs in use


The only way I've seen to stop the leak is to set debug.mpsafenet=0
It might be interesting to note that the number of packets dropped due
to full socket buffers is on the same order of magnitude as the number
of mbufs leaked:

% netstat -spudp
udp:
<...>
        3933 dropped due to full socket buffers


Other clues:

The leak is much worse with our device (MPSAFE, 4Gb/sec, up to 128 pkts/intr)
than with fxp.

I've also seen the leak with TCP, but only with our 4Gb/s device, not
with 100Mb/s fxp.  

So maybe there are 2 leaks, but I'd at least like to get the leak in
the system figured out before I try to find a leak caused by my driver ;)

Thanks,

Drew
Received on Thu Oct 14 2004 - 11:53:55 UTC

This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:38:17 UTC