Re: numa involved in instability and swap usage despite RAM free?

On 27/06/2018 07:52, Steve Kargl wrote:
> On Tue, Jun 26, 2018 at 02:39:27PM -0700, Adrian Chadd wrote:
>> On Mon, 25 Jun 2018 at 11:23, Steve Kargl
>> <sgk_at_troutmask.apl.washington.edu> wrote:
>>>
>>> On Sun, Jun 24, 2018 at 12:03:29PM +0200, Alexander Leidinger wrote:
>>>>
>>>> I don't have hard evidence, but there is enough "smell" to open up a
>>>> discussion...
>>>>
>>>> Short:
>>>> Can it be that enabling numa in the kernel is the reason why some
>>>> people see instability with zfs and usage of swap while a lot of free
>>>> RAM is available?
>>>
>>> Interesting observation.  I do have NUMA in my kernel, and swap
>>> seems to be used instead of recycling freeing inactive memory.
>>> Top shows
>>>
>>> Mem: 506M Active, 27G Inact, 98M Laundry, 2735M Wired, 1474M Buf, 1536M Free
>>> Swap: 16G Total, 120M Used, 16G Free

>From someone that has had memory issues since 10.1 (bug 194654), I have
recently realised something that seems to make some sense to me.

The arc_max setting is a limit of zfs arc and this ram gets wired to
prevent it swapping, this makes sense.

The vm.max_wired is a value that I had thought was ignored but now I see
that these are two values of wired memory which are not connected.
max_wired appears to default to 30% of kmem_size.

Both of these values are added together to be reported in
vm.stats.vm.v_wire_count which is the wired value shown by top. This
appears to be the reason that I can see 9G wired when max_wired is at 5G

The implications of this is that together (arc_max + max_wired) can be
set to more than the physical installed ram. I can verify that with 8G
installed and the two values add up to more than 7G you get no choice
but a hard reset. Since upgrading to 16G I have been more vigilant and
not allowed more than 10G to be wired so haven't had that problem in a
year and a half.

With the default arc_max usually set to ram minus 1G and max_wired at 5G
it is easy to see that the current defaults are dangerous.

I have not seen max_wired mentioned in relation to zfs but it seems that
it should be considered when setting arc_max to prevent over wiring ram.

Close to three weeks ago I applied review D7538 to my everyday desktop
running stable/11. Until a few days ago I had no swap usage which is now
at 9M. In the last few years of monitoring wired usage to try and find a
solution I have not seen less than 1G of swap usage after an hour of
uptime. If nothing else D7538 makes arc more willing to be released.


-- 
FreeBSD - the place to B...Storing Data

Shane Ambler