Re: VIMAGE: Freed UMA keg was not empty

"Bjoern A. Zeeb" <bzeeb-lists_at_lists.zabbadoz.net> a écrit
> On Wed, 17 Nov 2010, Thierry Herbelot wrote:
> 
> Hi,
> 
> first of all freebsd-virtualization_at_ is the better list for this; Cc:ed.

(in fact, I did not know where else to send this message : -net, ... ? thanks 
for the CC)
> 
> > We are using FreeBSD + VIMAGE at work, and we have seen an annoying
> > problem : there seems to be a memory leak in the kernel, which
> > eventually causes a panic.
> > 
> > (yes, we have seen the following message : "WARNING: VIMAGE (virtualized
> > network stack) is a highly experimental feature.")
> > 
> > Configuring a network interface in a jail with vnet enabled, then
> > removing that jail causes these messages. In dmesg:
> > 
> > [...]
> > Freed UMA keg was not empty (203 items).  Lost 1 pages of memory.
> > Freed UMA keg was not empty (36 items).  Lost 2 pages of memory.
> > 
> > The issue happens in a GENERIC FreeBSD 8.1 kernel, with VIMAGE enabled
> > (with attached VIMAGE kernel config file).
> > 
> > The following commands reproduce the bug:
> > 
> > jail -l -u root -c path=/ name=foo persist vnet &&
> > jexec foo ifconfig lo0 127.0.0.1/8 &&
> > jail -r foo
> > 
> > Running it too many times exhausts kernel memory and crashes the machine.
> > 
> > The probleme is seen on 8.1-release kernel, 8-stable from SVN and SVN
> > -head.
> 
> The problem has been present since day 1 and is still present up to
> HEAD.  This is about type stability and teardown.  So far we are no
> able to actually free TCP (and UDP in SVN) states as they might still
> be accesses after "free".  It's a general problem in the network stack
> and has been implemented as a measure to circumvent panics in those
> cases.
> 
> I am not sure if that's actually what's biting you wrt to memory or
> if it's something else.  Unfortunately boot logs and kernel configs
> don't help much; enable ddb and get show uma and show malloc reports
> after the crash (or watch vmstat -z and vmstat -m) after every 50 jail
> restarts.  I might have some patches for a couple of things but cannot
> (yet) help with the additional (duplicate) UMA zones showing up at
> each iteration (for the -z case).

The context for our problem is that VIMAGE jails are mant to be used "in 
production" for automated tests, thus are likely to be multiple on one 
physical host, and are likely to be started and stopped according to our 
needs.

We have tried more "classical" virtualization solutions (qemu, kvm, ... , 
normal jails with individual FIBs, ...) but only FreeBSD with VIMAGE seems to 
be the correct answer. (for other reasons, an i386 version of the kernel must 
be used)

The point of my original email was to simplify the configuration for a test 
setup : any machine with the attached configuration file and the above 
sequence of commands creating and deleting jails, running in a loop, will 
eventually panic.

Anyway, I will follow up with the logs you mentioned.

	Cheers (and thanks for the quick feedback)

	Thierry Herbelot
> 
> /bz