Re: stable/13, vm page counts do not add up

On Wed, Apr 07, 2021 at 10:42:57PM +0300, Andriy Gapon wrote:
> 
> I regularly see that the top's memory line does not add up (and by a lot).
> That can be seen with vm.stats as well.
> 
> For example:
> $ sysctl vm.stats | fgrep count
> vm.stats.vm.v_cache_count: 0
> vm.stats.vm.v_user_wire_count: 3231
> vm.stats.vm.v_laundry_count: 262058
> vm.stats.vm.v_inactive_count: 3054178
> vm.stats.vm.v_active_count: 621131
> vm.stats.vm.v_wire_count: 1871176
> vm.stats.vm.v_free_count: 187777
> vm.stats.vm.v_page_count: 8134982
> 
> $ bc
> >>> 187777 + 1871176 + 621131 + 3054178 + 262058
> 5996320
> >>> 8134982 - 5996320
> 2138662
> 
> As you can see, it's not a small number of pages either.
> Approximately 2 million pages, 8 gigabytes or 25% of the whole memory on this
> system.
> 
> This is 47c00a9835926e96, 13.0-STABLE amd64.
> I do not think that I saw anything like that when I used (much) older FreeBSD.

One relevant change is that vm_page_wire() no longer removes pages from
LRU queues, so the count of pages in the queues can include wired pages.
If the page daemon runs, it will dequeue any wired pages that are
encountered.

This was done to reduce queue lock contention, operations like
sendfile() which transiently wire pages would otherwise trigger two
queue operations per page.  Now that queue operations are batched this
might not be as important.

We could perhaps add a new flavour of vm_page_wire() which is not lazy
and would be suited for e.g., the buffer cache.  What is the primary
source of wired pages in this case?