Memory accounting discrepancy in top (possibly ZFS related)

From: Peter Schuller <peter.schuller_at_infidyne.com>
Date: Mon, 10 Sep 2007 22:40:45 +0200
Hello,

(I realize I am posting alot to -current lately. If people feel I am
too quick to flood the lists let me know, but I do try to keep to
things that seem relevant for -current.)

I have a machine (Dell 2950) where I orignally thought there was a
memory visibility problem. It has 4 GB in total which seemed to be
detected on boot based on dmesg, yet top showed a total of slightly
above 2 GB. I posted to -questions about this in the belief that this
was the case from boot onwards.

However, I discovered that after boot memory adds up in top, but then
goes down. Observe the following progression of top memory and their
totals over the course of about a day, on amd64 with CURRENT from
2007-09-09:

Mem: 10M Active, 1388K Inact, 84M Wired, 268K Cache, 800K Buf, 3822M Free
  Total: 3918
Mem: 272M Active, 31M Inact, 650M Wired, 42M Cache, 992K Buf, 1391M Free
  Total: 2386.9
Mem: 274M Active, 32M Inact, 655M Wired, 52M Cache, 992K Buf, 1372M Free
  Total: 2385.9
Mem: 282M Active, 31M Inact, 646M Wired, 53M Cache, 992K Buf, 1372M Free
  Total: 2384.9
Mem: 332M Active, 146M Inact, 719M Wired, 52M Cache, 214M Buf, 1071M Free
  Total: 2534

Possibly related is that on this machine I also recently saw a strange
situation where the amount of "inactive" memory was very high (2.5 gb)
when I would have expected 10-400 mb or something along those
lines. Even stranger is that if I then ran a memory grabbing process,
that memory would go from inactive to active, and then back to FREE
when the program terminated. This happened WITHOUT any disk I/O
(swapping). This was the reason why I cvsup:ed the machine on -09.

My understanding is that memory that is 'inactive' could only be freed
for use by another process by either the memory being de-allocated, or
going into swap. Thus I am/was unable to explain how the memory could
be pushed from inactive to free, by running a process that temporarily
gobbled up a few hundred megs of RAM, without seeing corresponding
disk I/O.

This situation occurred after I had dd:ed 8 gigabytes of zeros onto a
file on UFS, mdconfig attached it, and swapon:ed it. It may or may not
have been like this immediately before I did this; I do not know for
sure. This was after an otherwise fresh boot with nothing big running,
which is why the 2.5 GB seemed like a huge number.

loader.conf on this machine:

vm.kmem_size_max="1258291200"
vfs.zfs.prefetch_disable="1"
vfs.zfs.arc_max="838860800"

geom_label_load="YES"
geom_mirror_load="YES"

zfs_load="YES"
vfs.root.mountfrom="zfs:tank/root"

kern.ipc.semmni=256
kern.ipc.semmns=512
kern.ipc.semmnu=256

vmstat -m on the machine as it appears now with the top discrepancy,
but NOT necessarily matching the time of the unaccounted-for
"inactive" memory. Note that 'solaris' seems higher than it should be
given arc_max:

         Type InUse MemUse HighUse Requests  Size(s)
       DEVFS2   160    10K       -      568  16,32,64
   DEVFS_RULE    34    16K       -       34  64,512
        DEVFS    60     2K       -       61  16,128
  ntfs_nthash     1   512K       -        1  
    pfs_nodes    20     5K       -       20  256
         GEOM   256    44K       -   135264  16,32,64,128,256,512,1024,2048
      ata_dma     2     1K       -        2  256
       isadev     7     1K       -        7  128
   acd_driver     1     2K       -        1  2048
         cdev    23     6K       -       23  256
        sigio     1     1K       -        1  64
     filedesc   415   328K       -   130951  16,32,512,1024,2048,4096
         kenv    76    11K       -       82  16,32,64
       kqueue   120   135K       -   164250  256,2048
    proc-args   180     9K       -   166624  16,32,64,128,256
      ithread    96    18K       -       96  32,128,256
       prison     9     9K       -        9  16,64,2048
CAM dev queue     1     1K       -        1  128
       KTRACE   100    13K       -      100  128
    CAM queue     3     1K       -        3  16
       linker   125   260K       -      164  16,32,64,128,256,512,1024,2048
        lockf    44     6K       -  1473726  128
      acpisem    15     2K       -       15  128
       ip6ndp     4     1K       -        4  64,128
         temp    30   272K       -   186563  16,32,64,128,256,512,1024,2048,4096
       devbuf 16748 35139K       -    16847  16,32,64,128,256,512,1024,2048,4096
       module   376    47K       -      376  128
     mtx_pool     1     8K       -        1  
      subproc   854  1502K       -    88424  512,4096
         proc     2    16K       -        2  
      session    88    11K       -    18149  128
         pgrp   104    13K       -    21773  128
         cred   279    70K       -  2578543  256
      uidinfo    19     4K       -     7194  64,2048
       plimit    98    25K       -    87992  256
       kbdmux     6     9K       -        6  16,256,512,2048,4096
    sysctltmp     0     0K       -    55695  16,32,64,128
    sysctloid  4144   208K       -     4194  16,32,64,128
       sysctl     0     0K       -    34582  16,32,64
         umtx   592    74K       -      592  128
     p1003.1b     1     1K       -        1  16
         SWAP     2  1097K       -        2  64
       bus-sc    87    88K       -     3128  16,32,64,128,256,512,1024,2048,4096
          bus  1019    81K       -    22677  16,32,64,128,256,1024
      devstat    28    57K       -       28  32,4096
 eventhandler    77     7K       -       77  64,128
         kobj   235   940K       -     5123  4096
      md_disk     1     2K       -        1  2048
       mfibuf     6   139K       -       32  32,256,512,2048
         rman   114    14K       -      584  16,64,128
         sbuf     0     0K       -    39534  16,32,64,128,256,512,1024,2048,4096
      CAM SIM     1     1K       -        1  256
   CAM periph     2     1K       -        9  16,32,64,128,256
    taskqueue     9     1K       -        9  16,32,128
       Unitno     9     1K       -       55  32,64
          iov     0     0K       -   833411  16,64,128,256,512,1024,2048
     ioctlops     0     0K       -   341736  16,32,64,128,256,512,1024,2048,4096
          msg     4    30K       -        4  2048,4096
          sem     4    76K       -        4  
          shm     1    16K       -        1  
         ttys  3696   502K       -    16667  128,1024
         ptys    23     6K       -       23  256
      acpidev    54     4K       -       54  64
     mbuf_tag     0     0K       -  4322792  32,64
          pcb    50   154K       -    75751  16,32,64,128,4096
       soname    77     9K       -  1302904  16,32,128
       biobuf     0     0K       -        3  2048
     vfscache     1  1024K       -        1  
     vfs_hash     1   512K       -        1  
       vnodes    19     1K       -       78  32,256
  vnodemarker     0     0K       -   105700  512
        mount   198     8K       -      357  16,32,64,128,256
      CAM XPT    10     2K       -       25  32,64,128,1024
          BPF     3     1K       -        3  128
  ether_multi    15     1K       -       16  16,32,64
       ifaddr    55    17K       -       55  32,64,128,256,512,2048,4096
        ifnet     4     7K       -        4  256,2048
        clone     5    20K       -        5  4096
       arpcom     2     1K       -        2  16
           lo     1     1K       -        1  32
     routetbl    39     7K       -    23058  32,64,128,256,512
     in_multi     2     1K       -        2  64
    sctp_iter     0     0K       -        6  256
     sctp_ifn     2     1K       -        2  128
     sctp_ifa     7     1K       -        7  128
     sctp_vrf     1     1K       -        1  64
    sctp_a_it     0     0K       -        6  16
    hostcache     1    36K       -        1  
       tcplog     0     0K       -     3341  256
     syncache     1   100K       -        1  
    in6_multi    16     1K       -       16  32,64,128
     pci_link    16     2K       -       16  32,128
  nfss_daemon     1    16K       -        1  
audit_evclass   150     5K       -      187  32
       newblk     1     1K       -        1  512
     inodedep     1   512K       -        1  
      pagedep     1   128K       -        1  
  ufs_dirhash   771   150K       -      771  16,32,64,128,256,512
    ufs_mount     6    21K       -        6  512,2048,4096
      UMAHash     3    14K       -       12  512,1024,2048,4096
    vm_pgdata     2   129K       -        2  128
      memdesc     1     4K       -        1  4096
       acpica   843    74K       -    26775  16,32,64,128,256,512,1024
     acpitask     0     0K       -        1  64
      io_apic     2     4K       -        2  2048
  ata_generic     1     1K       -        1  1024
          msi     2     1K       -        2  128
     nexusdev     4     1K       -        4  16
      entropy  1024    64K       -     1024  64
     atkbddev     2     1K       -        2  64
       USBdev    21     8K       -       21  16,32,128,256,512
          USB    87    19K       -      102  16,32,64,128,256,1024
       DEVFS1   160    80K       -      164  512
       DEVFS3   917   230K       -      938  256
   zones_data     4     1K       -        4  16
      solaris 1163491 949421K       - 217472361  16,32,64,128,256,512,1024,2048,4096
   kstat_data     1     1K       -        1  64
  mirror_data     4     2K       -       31  64,256,512
        linux    14     1K       -       14  16,64


ZFS related sysctls (grep zfs):

vfs.zfs.arc_min: 39321600
vfs.zfs.arc_max: 838860800
vfs.zfs.mdcomp_disable: 0
vfs.zfs.prefetch_disable: 1
vfs.zfs.zio.taskq_threads: 0
vfs.zfs.recover: 0
vfs.zfs.vdev.cache.size: 10485760
vfs.zfs.vdev.cache.max: 16384
vfs.zfs.cache_flush_disable: 0
vfs.zfs.zil_disable: 0
vfs.zfs.debug: 0
kstat.zfs.misc.arcstats.hits: 12088667
kstat.zfs.misc.arcstats.misses: 769430
kstat.zfs.misc.arcstats.demand_data_hits: 3589717
kstat.zfs.misc.arcstats.demand_data_misses: 46936
kstat.zfs.misc.arcstats.demand_metadata_hits: 8498950
kstat.zfs.misc.arcstats.demand_metadata_misses: 722494
kstat.zfs.misc.arcstats.prefetch_data_hits: 0
kstat.zfs.misc.arcstats.prefetch_data_misses: 0
kstat.zfs.misc.arcstats.prefetch_metadata_hits: 0
kstat.zfs.misc.arcstats.prefetch_metadata_misses: 0
kstat.zfs.misc.arcstats.mru_hits: 3374213
kstat.zfs.misc.arcstats.mru_ghost_hits: 6831
kstat.zfs.misc.arcstats.mfu_hits: 8714454
kstat.zfs.misc.arcstats.mfu_ghost_hits: 66142
kstat.zfs.misc.arcstats.deleted: 576876
kstat.zfs.misc.arcstats.recycle_miss: 52143
kstat.zfs.misc.arcstats.mutex_miss: 1173
kstat.zfs.misc.arcstats.evict_skip: 61
kstat.zfs.misc.arcstats.hash_elements: 145695
kstat.zfs.misc.arcstats.hash_elements_max: 150128
kstat.zfs.misc.arcstats.hash_collisions: 2187962
kstat.zfs.misc.arcstats.hash_chains: 42687
kstat.zfs.misc.arcstats.hash_chain_max: 12
kstat.zfs.misc.arcstats.p: 479967938
kstat.zfs.misc.arcstats.c: 628830540
kstat.zfs.misc.arcstats.c_min: 39321600
kstat.zfs.misc.arcstats.c_max: 838860800
kstat.zfs.misc.arcstats.size: 628438016

So once again I don't have debugging information/traces available,
because I am not sure what would be wanted. If asked for I will
provide anything I can provide given the circumstances.

-- 
/ Peter Schuller

PGP userID: 0xE9758B7D or 'Peter Schuller <peter.schuller_at_infidyne.com>'
Key retrieval: Send an E-Mail to getpgpkey_at_scode.org
E-Mail: peter.schuller_at_infidyne.com Web: http://www.scode.org


Received on Mon Sep 10 2007 - 18:40:47 UTC

This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:39:17 UTC