Re: how to recycle Inact memory more aggressively?

On Tue, 15 Mar 2016 09:30:11 -0600, Ian Lepore <ian_at_freebsd.org> wrote:

> On Tue, 2016-03-15 at 07:20 -0700, Jeffrey Bouquet wrote:
> > rsync... see bottom posting
> > 
> > On Tue, 15 Mar 2016 07:43:46 +0100, olli hauer <ohauer_at_gmx.de> wrote:
> > 
> > > On 2016-03-14 15:19, Ian Lepore wrote:
> > > > On Sun, 2016-03-13 at 19:08 -0700, Adrian Chadd wrote:
> > > > > On 13 March 2016 at 18:51, Mark Johnston <markj_at_freebsd.org>
> > > > > wrote:
> > > > > > On Sun, Mar 13, 2016 at 06:33:46PM -0700, Adrian Chadd wrote:
> > > > > > > Hi,
> > > > > > > 
> > > > > > > I can reproduce this by doing a mkimage on a large
> > > > > > > destination
> > > > > > > file
> > > > > > > image. it looks like it causes all the desktop processes to
> > > > > > > get
> > > > > > > paged
> > > > > > > out whilst it's doing so, and then the whole UI freezes
> > > > > > > until it
> > > > > > > catches up.
> > > > > > 
> > > > > > mkimg(1) maps the destination file with MAP_NOSYNC, so if
> > > > > > it's
> > > > > > larger
> > > > > > than RAM, I think it'll basically force the pagedaemon to
> > > > > > write out
> > > > > > the
> > > > > > image as it tries to reclaim pages from the inactive queue.
> > > > > > This
> > > > > > can
> > > > > > cause stalls if the pagedaemon blocks waiting for some I/O to
> > > > > > complete.
> > > > > > The user/alc/PQ_LAUNDRY branch helps alleviate this problem
> > > > > > by
> > > > > > using a
> > > > > > different thread to launder dirty pages. I use mkimg on
> > > > > > various
> > > > > > desktop
> > > > > > machines to build bhyve images and have noticed the problem
> > > > > > you're
> > > > > > describing; PQ_LAUNDRY helps quite a bit in that case. But I
> > > > > > don't
> > > > > > know
> > > > > > why this would be a new problem.
> > > > > > 
> > > > > 
> > > > > That's why I'm confused. I just know that it didn't used to
> > > > > cause the
> > > > > whole UI to hang due to paging.
> > > > > 
> > > > 
> > > > I've been noticing this too.  This machine runs 10-stable and
> > > > this use
> > > > of the swap began happening recently when I updated from 10
> > > > -stable
> > > > around the 10.2 release time to 10-stable right about when the
> > > > 10.3
> > > > code freeze began.
> > > > 
> > > > In my case I have no zfs anything here.  I noticed the problem
> > > > bigtime
> > > > yesterday when rsync was syncing a ufs filesystem of about 500GB
> > > > from
> > > > one disk to another (probably 70-80 GB actually needed copying). 
> > > >  My
> > > > desktop apps were noticibly unresponsive when I activated a
> > > > window that
> > > > had been idle for a while (like it would take a couple seconds
> > > > for the
> > > > app to begin responding).  I could see lots of swap In happening
> > > > in top
> > > > during this unresponsiveness, and noticible amounts of Out
> > > > activity
> > > > when nothing was happening except the rsync.
> > > > 
> > > > This is amd64, 12GB ram, 16GB swap, a tmpfs had about 400MB in it
> > > > at
> > > > the time.  Prior to the update around the 10.3 freeze, this
> > > > machine
> > > > would never touch the swap no matter what workload I threw at it
> > > > (this
> > > > rsync stuff happens every day, it's the usual workload).
> > > > 
> > > 
> > > I'm not sure if it is the same problem, or port related.
> > > 
> > > On two systems without zfs but with many files e.g. svn servers I
> > > see now
> > > from time to time they are running out of swap.
> > > 
> > >  kernel: swap_pager_getswapspace(9): failed
> > >  kernel: swap_pager_getswapspace(16): failed
> > >  ...
> > > 
> > > It also happened on one system during the nightly periodic tasks
> > > holding
> > > only millions of backup files.
> > > 
> > > $ freebsd-version -ku
> > >   10.2-RELEASE-p9
> > >   10.2-RELEASE-p13
> > > 
> > > 
> > > _______________________________________________
> > > freebsd-current_at_freebsd.org mailing list
> > > https://lists.freebsd.org/mailman/listinfo/freebsd-current
> > > To unsubscribe, send any mail to "
> > > freebsd-current-unsubscribe_at_freebsd.org"
> > 
> > 
> > Just a point I've bought up elsewhere...
> > I've, if I recall, wrecked several filesystems (although EIDE) using
> > rsync at the normal bus rate, and sometimes
> > thumbdrives with whatever filesystem type on them.
> > 
> > I settled on --bwlimit=1500,  max for unattended  rsync usage and
> > almost every day
> > use --bwlimit=700.
> > 
> > The latter enables several resource-intensive processes ( music,
> > classical music videos, svn, pkg, browsing, etc) to
> > proceed apace concurrently on the desktop (SATA not EIDE) with nary a
> > hang nor slowdown.
> > 
> > If I recall, the usual speed is 10000 so that is less than ten
> > percent, if I recall, of the usual speed.
> > 
> > YMMV.
> > 
> > J.
> > 
> > PS as an afterthough, it would be useful if that were more prominent
> > on the man page somewhere or even
> > in the port's pkg-message or pkg-description.  
> > The SATA more robust than EIDE on FreeBSD that I've come across,
> > though I prefer not to hint at because I
> > believe it to be the fault of EIDE firmware rather than FreeBSD code.
> > FWIW.
> 
> I have no real idea what any of that is about, but before it turns into

Just trying to insert a helpful parameter into anyone's shell scripts

> some kind of "rsync is bad" mythology, let me just say that I've been

Quite not the point,  I apologize if that was the tone.

> using rsync to copy gigabytes of backup data every day for years now. 
>  I've never had any kind of problem, especially system responsiveness
> problems, until this increased swapfile activity thing started
> happening on 10-stable in the past few months.
> 
> To reiterate: rsync is not in any way at fault here, and any suggestion
> that the unresponsiveness should be "fixed" by screwing around with
> rsync parms that have worked fine for a decade is just something I
> completely reject.

I should maybe have put forth more prominently that the parameter was meant to
allow other resource-intensive processes to coexist, on a desktop machine.  
Apologies for the unintended oversight.

> 
> I'm sure I'd see the same kind of increased swapping with ANY process
> that read and wrote gigabytes of data in a short period of time.  And
> that's what this thread is about:  What has changed to cause this
> regression that multiple people are seeing where lots of IO now drives
> an unusual amount of swapfile activity on systems that used to NEVER
> write anything to swap?
> 
> -- Ian