Re: Trouble with recent auto-tuning changes

On Thu, 2013-01-31 at 18:13 +0100, Andre Oppermann wrote:
> On 28.01.2013 20:20, Alan Cox wrote:
> > On 01/28/2013 08:22, Ian Lepore wrote:
> >> On Mon, 2013-01-28 at 00:09 -0600, Alan Cox wrote:
> >>> On Sun, Jan 27, 2013 at 12:11 PM, Ian Lepore <ian_at_freebsd.org> wrote:
> >>>
> >>>> I ran into a panic while attempting to un-tar a large file on a
> >>>> DreamPlug (arm-based system) running -current.  The source and dest of
> >>>> the un-tar is the root filesystem on sdcard, and I get this:
> >>>>
> >>>> panic: kmem_malloc(4096): kmem_map too small: 12582912 total allocated
> >>>>
> >>>> Just before the panic I see the tar process get hung in a "nokva" wait.
> >>>> 12582912 is the value of VM_KMEM_SIZE from arm/include/vmparam.h.
> >>>>
> >>>> In r245575 the init order for mbuf limits was changed from
> >>>> SI_SUB_TUNABLES to SI_SUB_KMEM so that mbuf limits could be based on the
> >>>> results of sizing kernel memory.  Unfortunately, the process of sizing
> >>>> kernel memory relies on the mbuf limits; in kmeminit():
> >>>>
> >>>>          vm_kmem_size = VM_KMEM_SIZE + nmbclusters * PAGE_SIZE;
> >>>>
> >>>> Since r245575, nmbclusters is zero when this line of code runs.  If I
> >>>> manually plugin "32768" (the number tunable_mbinit() comes up with for
> >>>> this platform) in that line, the panic stops happening.
> >>>>
> >>>> So we've got two problems here... one is the circular dependency in
> >>>> calculating the mbuf limits.  The other is the fact that some
> >>>> non-trivial amount of kernel memory we're allowing for mbufs is actually
> >>>> being used for other things.  That is, if my system was actually using
> >>>> all the mbufs that tunable_mbinit() allowed for, then this panic while
> >>>> untarring a huge file would still have happened.
> >>>>
> >>>>
> >>> All of this is factually correct.  However, it's a red herring.  The real
> >>> problem is that arm, unlike every other architecture in the tree, does not
> >>> enable auto-sizing of the kmem map based on the physical memory size.
> >>> Specifically, you'll find VM_KMEM_SIZE_SCALE defined in
> >>> "arch"/include/vmparam.h on every other architecture, just not on arm.
> >>> This auto-sizing overrides the value of VM_KMEM_SIZE.
> >>>
> >> Aha.  I'll investigate what other architectures do with that and try to
> >> get the same thing going for arm.
> >>
> >
> > i386 or (32-bit) MIPS would be the most similar.  Also, I would
> > encourage you to look for other definitions that those architectures
> > have that arm doesn't.  As physical memory sizes continue to grow on
> > arm-based systems, they may require other changes in vmparam.h and the
> > machine-dependent param.h that were made on those other architectures
> > year ago.
> 
> Ian,
> 
> The patch below should do the trick.  Can you please test?

Yep, that fixed the problem with untarring the large file.  Here are
some before/after numbers from sysctl, converted from bytes to KBytes
for readability:

  vm.kmem_size_scale:       0         2
  vm.kmem_map_free:      5740    246440
  vm.kmem_map_size:      6548      7176
  vm.kmem_size:         12288    253616

  real memory  = 536870912 (512 MB)
  avail memory = 516718592 (492 MB)

-- Ian