Re: FreeBSD 5.1-Release freezes

On Thu, 26 Jun 2003, Socketd wrote:

> On Wed, 25 Jun 2003 16:44:56 -0400 (EDT)
> Robert Watson <rwatson_at_freebsd.org> wrote:
> 
> > The code most likely to cause a memory leak in the MAC Framework is
> > the label management code, since that's the only code that really does
> > much in the way of memory allocaiton.  Try compiling options MAC_DEBUG
> > into your kernel, which causes the MAC Framework to track the number
> > of labels it has allocated/free'd in a series of variables: 
> > 
> > static unsigned int nmacmbufs, nmaccreds, nmacifnets, nmacbpfdescs,
> >     nmacsockets, nmacmounts, nmactemp, nmacvnodes, nmacdevfsdirents,
> >     nmacipqs, nmacpipes, nmacprocs;
> 
> After running a few commands, ssh logins and loading mac_seeotheruids we
> now have:
<...>
> Loading mac_seeotheruids made vnodes increase a little, but it has (very
> slowly) been increasing and as you can see is now at 524 (don't know if
> this means anything).

You can compare the number of labeled vnodes with the number of actual
vnodes by looking at the vfs.numvnodes sysctl value -- ideally they should
be the same.

The fact that the problem went away with the addition of the debugging
code is a little concerning.  Here's what I'd like you to try, if you can:
set up a serial console to another machine.  Compile the kernel with
options DDB, but without options MAC_DEBUG.  Do whatever it is you do that
triggers the hang.  On the serial console, send a serial break and see if
you can get into DDB.  If you can, send the output of the following
commands:

  ps
  show lockedvnods

I may then ask you to run the trace command on various processes/threads,
and use "show locks" to show which threads hold which locks (assuming you
have WITNESS in the kernel, which also changes the timing).

On the whole, enabling MAC doesn't substantially change the locking, but
there are a couple of edge cases where we grab locks that were not
previously grabbed.  For example, we now grab vnode locks when polling a
vnode so we can do a label check on the vnode; this generates WITNESS
warnings as we hold the file descriptor lock over the call to VOP_POLL() 
(ouch).  If we wend up concluding this is a locking problem, it might be
that I send you some patches to modify the locking and see if we can get
rid of the problem.

thanks!

Robert N M Watson             FreeBSD Core Team, TrustedBSD Projects
robert_at_fledge.watson.org      Network Associates Laboratories