Re: stable/13, vm page counts do not add up

On Wed, Apr 07, 2021 at 11:22:41PM +0300, Andriy Gapon wrote:
> On 07/04/2021 22:54, Mark Johnston wrote:
> > On Wed, Apr 07, 2021 at 10:42:57PM +0300, Andriy Gapon wrote:
> >>
> >> I regularly see that the top's memory line does not add up (and by a lot).
> >> That can be seen with vm.stats as well.
> >>
> >> For example:
> >> $ sysctl vm.stats | fgrep count
> >> vm.stats.vm.v_cache_count: 0
> >> vm.stats.vm.v_user_wire_count: 3231
> >> vm.stats.vm.v_laundry_count: 262058
> >> vm.stats.vm.v_inactive_count: 3054178
> >> vm.stats.vm.v_active_count: 621131
> >> vm.stats.vm.v_wire_count: 1871176
> >> vm.stats.vm.v_free_count: 187777
> >> vm.stats.vm.v_page_count: 8134982
> >>
> >> $ bc
> >>>>> 187777 + 1871176 + 621131 + 3054178 + 262058
> >> 5996320
> >>>>> 8134982 - 5996320
> >> 2138662
> >>
> >> As you can see, it's not a small number of pages either.
> >> Approximately 2 million pages, 8 gigabytes or 25% of the whole memory on this
> >> system.
> >>
> >> This is 47c00a9835926e96, 13.0-STABLE amd64.
> >> I do not think that I saw anything like that when I used (much) older FreeBSD.
> > 
> > One relevant change is that vm_page_wire() no longer removes pages from
> > LRU queues, so the count of pages in the queues can include wired pages.
> > If the page daemon runs, it will dequeue any wired pages that are
> > encountered.
> 
> Maybe I misunderstand how that works, but I would expect that the sum of all
> counters could be greater than v_page_count at times.  But in my case it's less.

I misread, sorry.  You're right, what I described would cause double
counting.

I don't know what might be causing it then.  It could be a page leak.
The kernel allocates wired pages without adjusting the v_wire_count
counter in some cases, but the ones I know about happen at boot and
should not account for such a large disparity.  I do not see it on a few
systems that I have access to.

> > This was done to reduce queue lock contention, operations like
> > sendfile() which transiently wire pages would otherwise trigger two
> > queue operations per page.  Now that queue operations are batched this
> > might not be as important.
> > 
> > We could perhaps add a new flavour of vm_page_wire() which is not lazy
> > and would be suited for e.g., the buffer cache.  What is the primary
> > source of wired pages in this case?
> 
> It should be ZFS, I guess.
> 
> -- 
> Andriy Gapon