kmem_map auto-sizing and size dependencies

From: Andre Oppermann <andre_at_freebsd.org>
Date: Fri, 18 Jan 2013 16:29:43 +0100
The autotuning work is reaching into many places of the kernel and
while trying to tie up all lose ends I've got stuck in the kmem_map
and how it works or what its limitations are.

During startup the VM is initialized and an initial kernel virtual
memory map is setup in kmem_init() covering the entire KVM address
range.  Only the kernel itself is actually allocated within that
map.  A bit later on a number of other submaps are allocated (clean_map,
buffer_map, pager_map, exec_map).  Also in kmeminit() (in kern_malloc.c,
different from kmem_init) the kmem_map is allocated.

The (inital?) size of the kmem_map is determined by some voodoo magic,
a sprinkle of nmbclusters * PAGE_SIZE incrementor and lots of tunables.
However it seems to work out to an effective kmem_map_size of about 58MB
on my 16GB AMD64 dev machine:

vm.kvm_size: 549755809792
vm.kvm_free: 530233421824
vm.kmem_size: 16,594,300,928
vm.kmem_size_min: 0
vm.kmem_size_max: 329,853,485,875
vm.kmem_size_scale: 1
vm.kmem_map_size: 59,518,976
vm.kmem_map_free: 16,534,777,856

The kmem_map serves kernel malloc (via UMA), contigmalloc and everthing
else that uses UMA for memory allocation.

Mbuf memory too is managed by UMA which obtains the backing kernel memory
from the kmem_map.  The limits of the various mbuf memory types have
been considerably raised recently and may make use of 50-75% of all physically
present memory, or available KVM space, whichever is smaller.

Now my questions/comments are:

  Does the kmem_map automatically extend itself if more memory is requested?

  Should it be set to a larger initial value based on min(physical,KVM) space
  available?

  The use of nmbclusters for the initial kmem_map size calculation isn't
  appropriate anymore due to it being set up later and nmbclusters isn't the
  only mbuf relevant mbuf type.  We make significant use of page sized mbuf
  clusters too.

  The naming and output of the various vm.kmem_* and vm.kvm_* sysctls is
  confusing and not easy to reconcile.  Either we need some more detailing
  more aspects or less.  Plus perhaps sysctl subtrees to better describe the
  hierarchy of the maps.

  Why are separate kmem submaps being used?  Is it to limit memory usage of
  certain subsystems?  Are those limits actually enforced?

-- 
Andre
Received on Fri Jan 18 2013 - 14:29:47 UTC

This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:40:34 UTC