Re: Trouble with recent auto-tuning changes

On 28.01.2013 20:20, Alan Cox wrote:
> On 01/28/2013 08:22, Ian Lepore wrote:
>> On Mon, 2013-01-28 at 00:09 -0600, Alan Cox wrote:
>>> On Sun, Jan 27, 2013 at 12:11 PM, Ian Lepore <ian_at_freebsd.org> wrote:
>>>
>>>> I ran into a panic while attempting to un-tar a large file on a
>>>> DreamPlug (arm-based system) running -current.  The source and dest of
>>>> the un-tar is the root filesystem on sdcard, and I get this:
>>>>
>>>> panic: kmem_malloc(4096): kmem_map too small: 12582912 total allocated
>>>>
>>>> Just before the panic I see the tar process get hung in a "nokva" wait.
>>>> 12582912 is the value of VM_KMEM_SIZE from arm/include/vmparam.h.
>>>>
>>>> In r245575 the init order for mbuf limits was changed from
>>>> SI_SUB_TUNABLES to SI_SUB_KMEM so that mbuf limits could be based on the
>>>> results of sizing kernel memory.  Unfortunately, the process of sizing
>>>> kernel memory relies on the mbuf limits; in kmeminit():
>>>>
>>>>          vm_kmem_size = VM_KMEM_SIZE + nmbclusters * PAGE_SIZE;
>>>>
>>>> Since r245575, nmbclusters is zero when this line of code runs.  If I
>>>> manually plugin "32768" (the number tunable_mbinit() comes up with for
>>>> this platform) in that line, the panic stops happening.
>>>>
>>>> So we've got two problems here... one is the circular dependency in
>>>> calculating the mbuf limits.  The other is the fact that some
>>>> non-trivial amount of kernel memory we're allowing for mbufs is actually
>>>> being used for other things.  That is, if my system was actually using
>>>> all the mbufs that tunable_mbinit() allowed for, then this panic while
>>>> untarring a huge file would still have happened.
>>>>
>>>>
>>> All of this is factually correct.  However, it's a red herring.  The real
>>> problem is that arm, unlike every other architecture in the tree, does not
>>> enable auto-sizing of the kmem map based on the physical memory size.
>>> Specifically, you'll find VM_KMEM_SIZE_SCALE defined in
>>> "arch"/include/vmparam.h on every other architecture, just not on arm.
>>> This auto-sizing overrides the value of VM_KMEM_SIZE.
>>>
>> Aha.  I'll investigate what other architectures do with that and try to
>> get the same thing going for arm.
>>
>
> i386 or (32-bit) MIPS would be the most similar.  Also, I would
> encourage you to look for other definitions that those architectures
> have that arm doesn't.  As physical memory sizes continue to grow on
> arm-based systems, they may require other changes in vmparam.h and the
> machine-dependent param.h that were made on those other architectures
> year ago.

Ian,

The patch below should do the trick.  Can you please test?

> If you have any questions about any of the definitions, feel free to
> e-mail me.
>
> Alan
>
> P.S. When I get a little more free time, I intend to get in touch with
> Andre to address the apparent circular dependency.  For now just know
> that unless that circular dependency is combined with a lack of kmem map
> auto-sizing, like arm, it's basically harmless.

I'm working myself through it and will post a patch shortly that untangles
a lot of the obscure VM initialization stuff and moves it into the modern
SYSINIT world.

-- 
Andre

Index: arm/include/vmparam.h
===================================================================
--- arm/include/vmparam.h       (revision 246082)
+++ arm/include/vmparam.h       (working copy)
_at__at_ -134,13 +134,16 _at__at_
  #endif

  #define VM_MAX_KERNEL_ADDRESS  0xffffffff
+
  /*
   * Virtual size (bytes) for various kernel submaps.
   */
-
  #ifndef VM_KMEM_SIZE
-#define VM_KMEM_SIZE            (12*1024*1024)
+#define VM_KMEM_SIZE           (12*1024*1024)
  #endif
+#ifndef VM_KMEM_SIZE_SCALE
+#define VM_KMEM_SIZE_SCALE     (2)
+#endif

  #define MAXTSIZ        (16*1024*1024)
  #ifndef DFLDSIZ