Re: [CFT] ZFS v15 patch (version 3)

On 2010-Jul-08 23:30:33 +0200, Martin Matuska <mm_at_FreeBSD.org> wrote:
>On 8. 7. 2010 22:04, Peter Jeremy  wrote / napísal(a):
>> Without patching arc_memory_throttle(), a system behaves especially
>> poorly if it uses ZFS with any of mmap(2), UFS or NFS client - in my
>> case, ports/mail/mairix was almost guaranteed to wedge the system.
>> This is the problem that the following hack is intended to work around:
>>   perl -e '$x = "x" x 1000000;'
>>
>>   
>Regarding ARC, you might want to try the revision 209227 from head that
>is scheduled for MFC on 18.7.2010:
>http://people.freebsd.org/~mm/patches/zfs/head-12636.patch

I have done some testing with 8-STABLE with head-12636.patch and have
managed to successfully reproduce a deadlock.  The system is amd64
with 2GB RAM running a mixed UFS+ZFS environment.  On a freshly booted
system, I unmount/remount my ZFS /home and a UFS scratch filesystem
that contains a 1.5GB file [ensuring there is no cached data from
either FS].  I then dd(1) the 1.5GB UFS file to /dev/null and, once
that is finished, start mairix on my ~6GB mail directory (on ZFS
/home).  After some time, I get the following 'systat -v' output:

    4 users    Load  9.30  8.97  8.33                  Jul 14 09:49

Mem:KB    REAL            VIRTUAL                       VN PAGER   SWAP PAGER
        Tot   Share      Tot    Share    Free           in   out     in   out
Act  122308    4436   721892     7876   59824  count      
All  418376    7020 1074594k    38920          pages      
Proc:                                                            Interrupts
  r   p   d   s   w   Csw  Trp  Sys  Int  Sof  Flt        cow    4031 total
  4          76      133k    3  194   30  135             zfod        ata0 irq14
                                                          ozfod    30 bge0 irq16
99.8%Sys   0.2%Intr  0.0%User  0.0%Nice  0.0%Idle        %ozfod       atapci1 20
|    |    |    |    |    |    |    |    |    |    |       daefr       uhci0 ehci
==================================================        prcfr       uhci1 22
                                           dtbuf          totfr  2000 cpu0: time
Namei     Name-cache   Dir-cache    100000 desvn          react  2001 cpu1: time
   Calls    hits   %    hits   %       918 numvn          pdwak
                                       273 frevn          pdpgs
                                                          intrn
Disks   ad0   ad1                                  540404 wire
KB/t   0.00  0.00                                  297512 act
tps       0     0                                 1122808 inact
MB/s   0.00  0.00                                   57876 cache
%busy     0     0                                    1948 free
                                                   218192 buf

Apart from normal daemons, the only processes running are vmstat,
systat and mairix (via SSH sessions).  Note that the system is running
at virtually 100%sys with extremely low free memory and extremely high
context switches but no obviously useful activity.  At this stage, the
system is basically unusable (I can't even kill the mairix process).

My understanding of the problem is that the VM system sees "available"
RAM as the sum of "cache" and "free" - which is reasonably high so
there is no pressure to free up "inact" RAM.  OTOH, ZFS ARC only
counts "free" RAM - which is critically low so it throttles itself
but has no way to get the VM system to move RAM onto the "free" list.

-- 
Peter Jeremy