Re: VIMAGE: Freed UMA keg was not empty

Thierry Herbelot <thierry.herbelot_at_free.fr> a écrit
> "Bjoern A. Zeeb" <bzeeb-lists_at_lists.zabbadoz.net> a écrit
> 
> > On Wed, 17 Nov 2010, Thierry Herbelot wrote:
> > 
> > Hi,
> > 
> > first of all freebsd-virtualization_at_ is the better list for this; Cc:ed.
> 
> (in fact, I did not know where else to send this message : -net, ... ?
> thanks for the CC)
> 
> > > We are using FreeBSD + VIMAGE at work, and we have seen an annoying
> > > problem : there seems to be a memory leak in the kernel, which
> > > eventually causes a panic.
> > > 
> > > (yes, we have seen the following message : "WARNING: VIMAGE
> > > (virtualized network stack) is a highly experimental feature.")
> > > 
> > > Configuring a network interface in a jail with vnet enabled, then
> > > removing that jail causes these messages. In dmesg:
> > > 
> > > [...]
> > > Freed UMA keg was not empty (203 items).  Lost 1 pages of memory.
> > > Freed UMA keg was not empty (36 items).  Lost 2 pages of memory.
> > > 
> > > The issue happens in a GENERIC FreeBSD 8.1 kernel, with VIMAGE enabled
> > > (with attached VIMAGE kernel config file).
> > > 
> > > The following commands reproduce the bug:
> > > 
> > > jail -l -u root -c path=/ name=foo persist vnet &&
> > > jexec foo ifconfig lo0 127.0.0.1/8 &&
> > > jail -r foo
> > > 
> > > Running it too many times exhausts kernel memory and crashes the
> > > machine.
> > > 
> > > The probleme is seen on 8.1-release kernel, 8-stable from SVN and SVN
> > > -head.
> > 
> > The problem has been present since day 1 and is still present up to
> > HEAD.  This is about type stability and teardown.  So far we are no
> > able to actually free TCP (and UDP in SVN) states as they might still
> > be accesses after "free".  It's a general problem in the network stack
> > and has been implemented as a measure to circumvent panics in those
> > cases.
> > 
> > I am not sure if that's actually what's biting you wrt to memory or
> > if it's something else.  Unfortunately boot logs and kernel configs
> > don't help much; enable ddb and get show uma and show malloc reports
> > after the crash (or watch vmstat -z and vmstat -m) after every 50 jail
> > restarts.  I might have some patches for a couple of things but cannot
> > (yet) help with the additional (duplicate) UMA zones showing up at
> > each iteration (for the -z case).
> 
> The context for our problem is that VIMAGE jails are mant to be used "in
> production" for automated tests, thus are likely to be multiple on one
> physical host, and are likely to be started and stopped according to our
> needs.
> 
> We have tried more "classical" virtualization solutions (qemu, kvm, ... ,
> normal jails with individual FIBs, ...) but only FreeBSD with VIMAGE seems
> to be the correct answer. (for other reasons, an i386 version of the
> kernel must be used)
> 
> The point of my original email was to simplify the configuration for a test
> setup : any machine with the attached configuration file and the above
> sequence of commands creating and deleting jails, running in a loop, will
> eventually panic.
> 
> Anyway, I will follow up with the logs you mentioned.

As promised, here are the full logs (in attachment)

This is a serial console log showing the command loop that triggers the bug on 
a debug kernel and ensuing DDB session.

the obvious problem line is :
          routetbl         2684       303890K         3469

(further tests showed an increase of the routetbl malloc zone by 4MBytes for 
each vnet jail creation/destruction cycle)

	Cheers

	Thierry

> 
> 	Cheers (and thanks for the quick feedback)
> 
> 	Thierry Herbelot
> 
> > /bz
> 
> _______________________________________________
> freebsd-current_at_freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-current
> To unsubscribe, send any mail to "freebsd-current-unsubscribe_at_freebsd.org"