Re: get_swap_pager(x) failed

On Wed, Dec 13, 2017 at 10:34:02AM +0800, blubee blubeeme wrote:
> On Wed, Dec 13, 2017 at 5:53 AM, Mark Millard <markmi_at_dsl-only.net> wrote:
> 
> > blubee blubeeme gurenchan at gmail.com wrote on
> > Tue Dec 12 15:58:19 UTC 2017 :
> >
> > > On Tue, Dec 12, 2017 at 3:34 PM, blubee blubeeme <gurenchan at gmail.com
> > > >wrote:
> > > > I am seeing tons of these messages while running tail -f
> > /var/log/messages
> > > > ============
> > > > Dec 12 15:11:41 blubee kernel: swap_pager_getswapspace(25): failed
> > > . . .
> > > >  1159 blubee        5  20    0   149M 56876K select  6   1:05   0.00%
> > > > ibus-engine-chewing
> > > >
> > > > ===========
> > > >
> > > > What's with all the swap errors? I am running ZFS and I have 16GB of
> > ram,
> > > > how could I be having swap space errors?
> > > >
> > >
> > > Well I added 4GB of extra swap in /var/tmp/swap0
> > > then added that to my /etc/fstab: md99                    none    swap
> > > sw,file=/var/tmp/swap0,late     0       0
> > >
> > > and those errors went away.
> >
> > I recommend reviewing bugzilla 206048 (title in part
> > "swapfile usage hangs; swap partition works"):
> >
> > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=206048
> >
> > before using file-system based swap spaces. They have
> > lots of problems with deadlocks. See especially comments
> > #7 and #8 quoting Konstantin Belousov. #8 is just a
> > reference to:
> >
> > https://www.freebsd.org/doc/en_US.ISO8859-1/books/developers-handbook/
> > kerneldebug-deadlocks.html
> >
> > Comment #3 shows a way to test for the problematical
> > behavior.
> >
> > Using swap partitions avoids the issue.
> >
> > ===
> > Mark Millard
> > markmi at dsl-only.net
> >
> >
> Thanks for the info, why would I be getting swap errors like that when I
> have 32GB of ram?
> sysctl hw.physmem
> hw.physmem: 34253692928
> 
> That really doesn't make any sense to me... Is it Chromium eating up 32GB+
> of ram?

Your top output from the first message in the thread gives a hint:

Mem: 3442M Active, 293M Inact, 4901M Laundry, 22G Wired, 1057M Free             
ARC: 18G Total, 1488M MFU, 15G MRU, 4003K Anon, 178M Header, 1171M Other        
     16G Compressed, 23G Uncompressed, 1.41:1 Ratio                             
Swap: 2048M Total, 2028M Used, 20M Free, 99% Inuse, 36K In                      

You have 22GB RAM used by the kernel, and 18GB of that is used by ZFS.
As you can see from the last line, 99% of your small swap space is used
and paging was happening at the time of the snapshot ("36K In").

First I would suggest limiting ARC and seeing if that helps.  Unless you
are doing a lot of fileserver type work or working with lots of files
you want to keep around that ARC size is a bit big.  My desktop with
32GB RAM has ARC limited to 4GB.

Definitely do NOT swap to a file on a filesystem, especially if the
filesystem is ZFS.  That will eventually lead to a deadlock.

An open question would be why ARC is not reducing if the system is
under memory pressure.  It's meant to, but there have been various
bugs in that implementation.

Regards,

Gary