Re: Trouble with recent auto-tuning changes

On 31.01.2013 23:25, Ian Lepore wrote:
> On Thu, 2013-01-31 at 18:13 +0100, Andre Oppermann wrote:
>> On 28.01.2013 20:20, Alan Cox wrote:
>>> On 01/28/2013 08:22, Ian Lepore wrote:
>>>> On Mon, 2013-01-28 at 00:09 -0600, Alan Cox wrote:
>>>>> On Sun, Jan 27, 2013 at 12:11 PM, Ian Lepore <ian_at_freebsd.org> wrote:
>>>>>
>>>>>> I ran into a panic while attempting to un-tar a large file on a
>>>>>> DreamPlug (arm-based system) running -current.  The source and dest of
>>>>>> the un-tar is the root filesystem on sdcard, and I get this:
>>>>>>
>>>>>> panic: kmem_malloc(4096): kmem_map too small: 12582912 total allocated
>>>>>>
>>>>>> Just before the panic I see the tar process get hung in a "nokva" wait.
>>>>>> 12582912 is the value of VM_KMEM_SIZE from arm/include/vmparam.h.
>>>>>>
>>>>>> In r245575 the init order for mbuf limits was changed from
>>>>>> SI_SUB_TUNABLES to SI_SUB_KMEM so that mbuf limits could be based on the
>>>>>> results of sizing kernel memory.  Unfortunately, the process of sizing
>>>>>> kernel memory relies on the mbuf limits; in kmeminit():
>>>>>>
>>>>>>           vm_kmem_size = VM_KMEM_SIZE + nmbclusters * PAGE_SIZE;
>>>>>>
>>>>>> Since r245575, nmbclusters is zero when this line of code runs.  If I
>>>>>> manually plugin "32768" (the number tunable_mbinit() comes up with for
>>>>>> this platform) in that line, the panic stops happening.
>>>>>>
>>>>>> So we've got two problems here... one is the circular dependency in
>>>>>> calculating the mbuf limits.  The other is the fact that some
>>>>>> non-trivial amount of kernel memory we're allowing for mbufs is actually
>>>>>> being used for other things.  That is, if my system was actually using
>>>>>> all the mbufs that tunable_mbinit() allowed for, then this panic while
>>>>>> untarring a huge file would still have happened.
>>>>>>
>>>>>>
>>>>> All of this is factually correct.  However, it's a red herring.  The real
>>>>> problem is that arm, unlike every other architecture in the tree, does not
>>>>> enable auto-sizing of the kmem map based on the physical memory size.
>>>>> Specifically, you'll find VM_KMEM_SIZE_SCALE defined in
>>>>> "arch"/include/vmparam.h on every other architecture, just not on arm.
>>>>> This auto-sizing overrides the value of VM_KMEM_SIZE.
>>>>>
>>>> Aha.  I'll investigate what other architectures do with that and try to
>>>> get the same thing going for arm.
>>>>
>>>
>>> i386 or (32-bit) MIPS would be the most similar.  Also, I would
>>> encourage you to look for other definitions that those architectures
>>> have that arm doesn't.  As physical memory sizes continue to grow on
>>> arm-based systems, they may require other changes in vmparam.h and the
>>> machine-dependent param.h that were made on those other architectures
>>> year ago.
>>
>> Ian,
>>
>> The patch below should do the trick.  Can you please test?
>
> Yep, that fixed the problem with untarring the large file.  Here are
> some before/after numbers from sysctl, converted from bytes to KBytes
> for readability:
>
>    vm.kmem_size_scale:       0         2
>    vm.kmem_map_free:      5740    246440
>    vm.kmem_map_size:      6548      7176
>    vm.kmem_size:         12288    253616
>
>    real memory  = 536870912 (512 MB)
>    avail memory = 516718592 (492 MB)

Thank you for testing.  Committed as r246204.  If any other problems come
up please report.

-- 
Andre