Re: FreeBSD 5.1-R kernel panic

Stephane Raimbault wrote:
> Well I went to go change my /boot/loader.conf options to reflect the
> following:
> 
> kern.vm.kmem.size="350000"

Assuming this is in pages, it is 1/3 of the total physical RAM in the
system.  This is way too large, unless you have recompiled your kernel
to have 3G KVA vs. 1G UVA, instead of the default.  Kernel recompilation
is required because the load address is a fixed virtual address based on
the KVA size, subtracted from the top of memory.  This is so that user
programs do not have to be recompiled when you move the ratio of UVA to
KVA around (with the exception of the Linux emulator, if you are using
Linux threads, since it seems to want its threads mailboxes at a known
location in memory).

> kern.ipc.nmbclusters="8192"

This is large, as well.

You need to turn off automatic tuning, and act like you are on a
memory budget equal to the amount of KVA space in the system,
which you can use for different things, but not at the same time.

If you leave the system at the defaults, it will (mostly) be able
to keep itself under the budget ceiling for an assumed 2G KVA
space, but as soon as you start tuning some things up, you will
very quickly blow your budget, since you tuning something you want
to use up will not necessarily tune something you aren't going to
use down, to avoid blowing your total budget.


> and enabled "options DDB" in my kernel.
> 
> Unfortunately, I ran into a problem on the reboot, the SMP kernel would fail
> to load due to some of my /boot/loader.conf changes.... perhaps I
> miss-understood something in your instructions... anyhow, this is what was
> displayed on the console when I set those above settings in the
> /boot/loader.conf... and since I had the debugger running, I did a trace and
> included in the paste below... is that the kind of stuff you will want if
> the box crashes again as originally mentioned in this thread.

This particular panic is an initialization of a region with a
header structure, where the memory allocation has failed because
you have already blown your budget, and so the allocation returns
NULL.

This happens early enough in the system startup that there is no
error checking to ensure that the allocation has succeeded.  But
even if you added error checking in all these places, the best the
system is going to be able to do is pnaic with a slightly more
informative message.

When it goes to fill in this structure, it tries to fill in a
field containing something 8 bytes into the structure, and
explodes.  If you look at the function where the traceback says
it crashed, this should be visibly obvious to you.


A good rule of thumb for tuning is to start with the autotuned
defaults (though they can screw you on occasion, since RAM is
installed in discrete amounts, and the autotuning tends to use
a continuous function of the amount of physical RAM, so you get
a "stair function", and not all possible steps in the stair have
actually been tested with all possible resource allocations having
been made, to see if a problem occurs).

Once you know those, you hard-code them so that they are no longer
autotuned.

Then you tweak the resource allocations for resources you don't
care about using down.

Then you tweak what you do care about using up, until you crash.

Then you back off on your last tweak to give you some safety
margin.

This will get you within 85%-90% of where you can get without
modifying code.


The only other alternative is to know where every byte of memory
is going.

-- Terry