FreeBSD CURRENT buildworld in virtualbox guest crashes FreeBSD host

From: Don Lewis <truckman_at_FreeBSD.org>
Date: Wed, 25 Sep 2019 15:52:08 -0700 (PDT)
Please forgive the crosspost, I don't know where the problem actually
belongs.

For the last several months I've had a problem where doing a buildworld
on FreeBSD CURRENT running as a virtualbox guest is crashing a the
FreeBSD current host.

This doesn't always happen, but there is a pattern.  The host in
question is primarily used for building packages sets with poudriere. If
I do a poudriere run, and after poudriere is finished, I start the
virtual box guest and do a buildworld there, this host will panic,
usually after an hour or so.  After I reboot the host and restart the
guest, the guest's UFS filesystem has some pretty severe damage, which
is not terribly unexpected.  I've never had the host panic under the
same circumstances if this host is freshly booted, though I did
experience a panic this morning during the subsequent installworld.  I'm
guessing that whether the panic happens or not depends on how much the
host memory map has been churned since boot.  According to top, there
are always around 5 GB of free memory when the crash occurs.

As mentioned, the problem has been happening with multiple revisions of
-CURRENT over the last several months, as well as multiple versions of
virtualbox.  This morning's crash happened with:
  FreeBSD zipper.catspoiler.org 13.0-CURRENT FreeBSD 13.0-CURRENT r352198 GENERIC  amd64
  virtualbox-ose-5.2.32_1        General-purpose full virtualizer for x86 hardware
  virtualbox-ose-kmod-5.2.32_1   VirtualBox kernel module for FreeBSD

The CPU is:
CPU: AMD Ryzen 7 1700X Eight-Core Processor          (3393.72-MHz K8-class CPU)
  Origin="AuthenticAMD"  Id=0x800f11  Family=0x17  Model=0x1  Stepping=1
  Features=0x178bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,C
MOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2,HTT>
  Features2=0x7ed8320b<SSE3,PCLMULQDQ,MON,SSSE3,FMA,CX16,SSE4.1,SSE4.2,MOVBE,POP
CNT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND>
  AMD Features=0x2e500800<SYSCALL,NX,MMX+,FFXSR,Page1GB,RDTSCP,LM>
  AMD Features2=0x35c233ff<LAHF,CMP,SVM,ExtAPIC,CR8,ABM,SSE4A,MAS,Prefetch,OSVW,
SKINIT,WDT,TCE,Topology,PCXC,PNXC,DBE,PL2I,MWAITX>
  Structured Extended Features=0x209c01a9<FSGSBASE,BMI1,AVX2,SMEP,BMI2,RDSEED,AD
X,SMAP,CLFLUSHOPT,SHA>
  XSAVE Features=0xf<XSAVEOPT,XSAVEC,XINUSE,XSAVES>
  AMD Extended Feature Extensions ID EBX=0x1007<CLZERO,IRPerf,XSaveErPtr,IBPB>
  SVM: NP,NRIP,VClean,AFlush,DAssist,NAsids=32768
  TSC: P-state invariant, performance statistics

and the machine has 64 GB of ECC RAM.

The stack trace is wierd:

zipper.catspoiler.org dumped core - see /var/crash/vmcore.7

Fri Aug  9 18:03:07 PDT 2019

FreeBSD zipper.catspoiler.org 13.0-CURRENT FreeBSD 13.0-CURRENT r350665 GENERIC  amd64

panic: page fault

GNU gdb (GDB) 8.3 [GDB v8.3 for FreeBSD]
Copyright (C) 2019 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-portbld-freebsd13.0".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /boot/kernel/kernel...
Reading symbols from /usr/lib/debug//boot/kernel/kernel.debug...

Unread portion of the kernel message buffer:


Fatal trap 12: page fault while in kernel mode
cpuid = 7; apic id = 07
fault virtual address	= 0xd45647000
fault code		= supervisor write data, page not present
instruction pointer	= 0x20:0xffffffff82f81b35
stack pointer	        = 0x28:0xfffffe01862681f0
frame pointer	        = 0x28:0xfffffe0186268250
code segment		= base 0x0, limit 0xfffff, type 0x1b
			= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags	= interrupt enabled, resume, IOPL = 0
current process		= 34192 (VirtualBox)
trap number		= 12
panic: page fault
cpuid = 7
time = 1565393713
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe0186267eb0
vpanic() at vpanic+0x19d/frame 0xfffffe0186267f00
panic() at panic+0x43/frame 0xfffffe0186267f60
trap_fatal() at trap_fatal+0x39c/frame 0xfffffe0186267fc0
trap_pfault() at trap_pfault+0x62/frame 0xfffffe0186268010
trap() at trap+0x2b4/frame 0xfffffe0186268120
calltrap() at calltrap+0x8/frame 0xfffffe0186268120
--- trap 0xc, rip = 0xffffffff82f81b35, rsp = 0xfffffe01862681f0, rbp = 0xfffffe0186268250 ---
ng_ether_ifnet_arrival_cookie() at 0xffffffff82f81b35/frame 0xfffffe0186268250
ng_ether_ifnet_arrival_cookie() at 0xffffffff82f695c5/frame 0xfffffe01862682e0
ng_ether_ifnet_arrival_cookie() at 0xffffffff82f682fb/frame 0xfffffe0186268350
ng_ether_ifnet_arrival_cookie() at 0xffffffff82ec5f2c/frame 0xfffffe0186268380
ng_ether_ifnet_arrival_cookie() at 0xffffffff82ec461f/frame 0xfffffe0186268400
ng_ether_ifnet_arrival_cookie() at 0xffffffff82ec0ec3/frame 0xfffffe0186268490
ng_ether_ifnet_arrival_cookie() at 0xffffffff82f88a36/frame 0xfffffe0186268530
ng_ether_ifnet_arrival_cookie() at 0xffffffff82ec7e8f/frame 0xfffffe01862685b0
supdrvIOCtlFast() at supdrvIOCtlFast+0xb8/frame 0xfffffe01862685d0
VBoxDrvFreeBSDIOCtl() at VBoxDrvFreeBSDIOCtl+0x5c/frame 0xfffffe0186268630
VBoxDrvFreeBSDIOCtlSMAP() at VBoxDrvFreeBSDIOCtlSMAP+0x31/frame 0xfffffe01862686b0
devfs_ioctl() at devfs_ioctl+0xca/frame 0xfffffe0186268700
VOP_IOCTL_APV() at VOP_IOCTL_APV+0x63/frame 0xfffffe0186268720
vn_ioctl() at vn_ioctl+0x13d/frame 0xfffffe0186268830
devfs_ioctl_f() at devfs_ioctl_f+0x1f/frame 0xfffffe0186268850
kern_ioctl() at kern_ioctl+0x27e/frame 0xfffffe01862688b0
sys_ioctl() at sys_ioctl+0x15d/frame 0xfffffe0186268980
amd64_syscall() at amd64_syscall+0x2bb/frame 0xfffffe0186268ab0
fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe0186268ab0
--- syscall (54, FreeBSD ELF64, sys_ioctl), rip = 0x80056c44a, rsp = 0x7fffde8dae38, rbp = 0x7fffde8dae40 ---
KDB: enter: panic

__curthread () at /usr/src/sys/amd64/include/pcpu.h:246
246		__asm("movq %%gs:%P1,%0" : "=r" (td) : "n" (OFFSETOF_CURTHREAD));
(kgdb) #0  __curthread () at /usr/src/sys/amd64/include/pcpu.h:246
#1  doadump (textdump=-2119273992) at /usr/src/sys/kern/kern_shutdown.c:392
#2  0xffffffff8049e77c in db_fncall_generic (addr=<optimized out>, 
    args=<optimized out>, rv=<optimized out>, nargs=<optimized out>)
    at /usr/src/sys/ddb/db_command.c:610
#3  db_fncall (dummy1=<optimized out>, dummy2=<optimized out>, 
    dummy3=<optimized out>, dummy4=<optimized out>)
    at /usr/src/sys/ddb/db_command.c:658
#4  0xffffffff8049e2b9 in db_command (last_cmdp=<optimized out>, 
    cmd_table=<optimized out>, dopager=1) at /usr/src/sys/ddb/db_command.c:482
#5  0xffffffff8049e034 in db_command_loop ()
    at /usr/src/sys/ddb/db_command.c:535
#6  0xffffffff804a11df in db_trap (type=<optimized out>, code=<optimized out>)
    at /usr/src/sys/ddb/db_main.c:252
#7  0xffffffff80c17f3c in kdb_trap (type=3, code=0, tf=<optimized out>)
    at /usr/src/sys/kern/subr_kdb.c:692
#8  0xffffffff810a9bc1 in trap (frame=0xfffffe0186267de0)
    at /usr/src/sys/amd64/amd64/trap.c:621
#9  <signal handler called>
#10 kdb_enter (why=0xffffffff8133d7ef "panic", msg=<optimized out>)
    at /usr/src/sys/kern/subr_kdb.c:479
#11 0xffffffff80bcd5da in vpanic (fmt=<optimized out>, ap=<optimized out>)
    at /usr/src/sys/kern/kern_shutdown.c:894
#12 0xffffffff80bcd353 in panic (
    fmt=0xffffffff81e88a98 <cnputs_mtx> "x&0\201\377\377\377\377")
    at /usr/src/sys/kern/kern_shutdown.c:832
#13 0xffffffff810aa01c in trap_fatal (frame=0xfffffe0186268130, 
    eva=56998785024) at /usr/src/sys/amd64/amd64/trap.c:943
#14 0xffffffff810aa082 in trap_pfault (frame=0xfffffe0186268130, 
    usermode=<optimized out>) at /usr/src/sys/amd64/amd64/trap.c:767
#15 0xffffffff810a9664 in trap (frame=0xfffffe0186268130)
    at /usr/src/sys/amd64/amd64/trap.c:443
#16 <signal handler called>
#17 0xffffffff82f81b35 in ?? ()
#18 0xfffffe0005c64dc0 in ?? ()
#19 0xfffffe00b7941000 in ?? ()
#20 0xfffffe0186268220 in ?? ()
#21 0x0000000000000046 in ?? ()
#22 0x0000000000000001 in ?? ()
#23 0x0002eccf00000000 in ?? ()
#24 0xfffffe01000028e3 in ?? ()
#25 0x00000005f1a00000 in ?? ()
#26 0x000000000000018d in ?? ()
#27 0xfffffe00b7971000 in ?? ()
#28 0xfffffe00b7941000 in ?? ()
#29 0x00000005f1a00000 in ?? ()
#30 0xfffffe01862682e0 in ?? ()
#31 0xffffffff82f695c5 in ?? ()
#32 0x000000000000018d in ?? ()
#33 0x0000000000000000 in ?? ()
(kgdb) 

The ng_ether stuff is something that the debuggers can't understand
because the code is something that virtualbox loads itself from
/usr/local/lib/virtualbox/VMMR0.r0.  If I dig through this manually
(from a different crash), I think that part of the stack trace should
actually be:

FFFFFFFF82F74B35        DFB15   pgmPoolAlloc+0x705
FFFFFFFF82F5C5C5        C75A5   pgmR0BthAMD64ProtTrap0eHandler+0x1375
FFFFFFFF82F5B2FB        C62DB   pgmR0BthAMD64ProtTrap0eHandler+0xab
FFFFFFFF82EB8F2C        23F0C   PGMR0Trap0eHandlerNestedPaging+0x5c
FFFFFFFF82EB761F        225FF   SVMR0RunGuestCode+38DF
FFFFFFFF82EB3EC3        1EEA3   SVMR0RunGuestCode+0x183
FFFFFFFF82F7BA36        E6A16   vmmR0CallRing3SetJmp_EndProc+0xbc
FFFFFFFF82EBAE8F        25E6F   pfnVMMR0EntryFast+0x57F

I'm not certain, but I think the crash is happening in the
ASMMemZeroPage() call here
(VirtualBox-5.2.20/src/VBox/VMM/VMMAll/PGMAllPool.cpp line 5277):

    if (!pPage->fZeroed)
    {
        STAM_PROFILE_START(&pPool->StatZeroPage, z);
        void *pv = PGMPOOL_PAGE_2_PTR(pVM, pPage);
        ASMMemZeroPage(pv);
        STAM_PROFILE_STOP(&pPool->StatZeroPage, z);
    }


------------------------------------------------------------------------
vmstat -m

         Type InUse MemUse HighUse Requests  Size(s)
      CAM XPT    41     4K       -      135  16,32,64,128,512,1024,2048,65536
      entropy     1     1K       -   968483  16,32,64,128,256,1024,4096
      CAM DEV    13    26K       -       48  2048
      CAM CCB     0     0K       - 37030322  2048
     CAM path    15     1K       -      136  32
   CAM periph     6     2K       -       38  16,32,64,128,256
    acpi_perf    16     2K       -       16  128
         hdaa    26   111K       -       26  256,512,1024,2048,16384,32768
         hdac     2     2K       -        2  512,1024
        hdacc     6     1K       -        6  32
       feeder    30     4K       -       40  32,128
        mixer     8    32K       -        8  4096
CAM I/O Scheduler     2     1K       -        2  128
    CAM queue    24     7K       -      131  16,32,512
         UART    12    13K       -       12  16,1024,2048
          USB    18    15K       -       18  16,128,512,4096
       USBdev    15     3K       -       15  32,64,128,256,512
CAM dev queue    11     1K       -       11  64
        vtbuf    24  1968K       -       46  4096
           vt    11     6K       -       11  512
       DEVFS2   126    32K       -     3240  16,32,64,128,256
       DEVFS3  1655   414K       -    11863  256
       DEVFS1   129   129K       -      139  1024
   DEVFS_RULE    56    27K       -       56  64,512
        DEVFS   123     4K       -      872  16,32,128
       DEVFSP     1     1K       -        2  64
NFSD V4client     1     1K       -        1  256
 NFSD lckfile     1     1K       -        1  256
       NFS fh   123     4K       -     3397  32
 NFSD session     1     1K       -        1  1024
    newnfsmnt     2     4K       -        2  2048
    pfs_nodes    83    42K       -       83  512
  pfs_vncache     5     1K       -     1125  64
         GEOM   104    18K       -      756  16,32,64,128,256,512,1024,2048,4096,8192,16384
    raid_data     0     0K       -       90  32,128,256
       isadev     8     1K       -        8  128
  ddb_capture     1    64K       -        1  65536
         cdev     3     1K       -        3  256
     filedesc     8    33K       -   429407  16,32,64,4096,8192,16384,32768
        sigio     1     1K       -        3  64
     filecaps     0     0K       -  3331075  16,32,64
      kdtrace   823   187K       - 56812404  64,256
         kenv   111    13K       -      114  16,32,64,128,8192
       kqueue   130    35K       - 27919565  64,256,512,2048
    proc-args   190     8K       - 15255055  16,32,64,128,256
        hhook    13     4K       -       13  256
      ithread   212    36K       -      212  32,128,256
       prison    44    44K       -      452  16,32,256,4096
       KTRACE   100    13K       -      100  128
       linker   409   619K       -      486  16,32,64,128,256,512,1024,2048,4096,8192,16384,32768,65536
        lockf    54     6K       - 13017741  64,128
   loginclass     3     1K       -        3  64
        cache     1     1K       -        1  32
       devbuf 17663 37236K       -    17721  16,32,64,128,256,512,1024,2048,4096,8192,65536
         temp    71    34K       - 120862207  16,32,64,128,256,512,1024,2048,4096,8192,16384,32768,65536
        evdev     3     3K       -        3  1024
       module   510    64K       -      511  128
     mtx_pool     2    72K       -        2  8192,65536
          osd    18     2K       -    55323  16,32,64,128,256
     pmchooks     1     1K       -        1  128
          pmc     1     1K       -        1  64
         pgrp    58     8K       -   407286  128
      session    53     7K       -     1105  128
         proc     3   264K       -        3  8192
      subproc  1058   897K       - 27787408  16,512,4096
         cred   159    40K       -  2247530  256
     acpiintr     1     1K       -        1  64
       plimit    26     7K       -  2202009  256
      uidinfo     8    33K       -     1164  128,32768
       dumper     2     1K       -        2  128,512
       acpica  9268   945K       -   336425  16,32,64,128,256,512,1024,2048,4096
       sysctl     0     0K       -   645696  64
    sysctloid  8626   438K       -     8796  16,32,64,128
    sysctltmp     0     0K       -  1331754  16,32,64,128,256,1024
     acpitask     1    64K       -        1  65536
      tidhash     1   256K       -        1  
      callout    17  5256K       -       17  
         umtx  4284   536K       -     4284  128
     p1003.1b     1     1K       -        1  16
         SWAP     1  2604K       -        1  
          bus  1561   143K       -    38138  16,32,64,128,256,512,1024,4096
       bus-sc   135  4907K       -     7531  16,32,64,128,256,512,1024,2048,4096,8192,16384,32768,65536
       kbdmux     6    22K       -        6  16,512,1024,2048,16384
      devstat     6    13K       -        6  32,4096
        epoch  2147    68K       -     2147  32,128
 eventhandler   133    11K       -      133  64,128
   gtaskqueue   106    43K       -      106  16,32,256,8192
         kobj   339  1356K       -      513  4096
      Per-cpu     1     1K       -        1  32
         rman   346    39K       -      763  16,32,128
         sbuf     0     0K       -    49365  16,32,64,128,256,512,1024,2048,4096,8192,16384,32768,65536
    toponodes    43     6K       -       43  128
    taskqueue   144    18K       -      159  16,32,64,128,256
     terminal    11     3K       -       11  256
       Unitno    36     2K       -  2520104  32,64
         vmem     3  1089K       -        8  512,16384,32768,65536
     ioctlops     0     0K       -     6203  256,512,1024
       select  1530   192K       -     1530  128
          iov     0     0K       - 314684863  16,64,128,256,512,1024,4096,16384,32768
          msg     4    30K       -        4  2048,4096,8192,16384
          sem     4   106K       -        4  2048,4096
          shm     3    36K       -       32  2048,32768
          tty    20    20K       -       25  1024
          pts     4     1K       -        9  256
     mbuf_tag     0     0K       -       25  32
        shmfd     1     8K       -      136  256,1024,8192
       soname    35     2K       -  3221619  16,32,64,128
          pcb    39  5274K       -     1605  16,32,64,128,256,1024,2048,8192
          acl     0     0K       -  3202318  4096
     vfscache     4 16577K       -        4  256,65536
     vfs_hash     1  8192K       -        1  
       vnodes     1     1K       -    28390  32,256
        mount  1162    46K       -  1122566  16,32,64,128,256,1024
       statfs     0     0K       - 23975531  4096
  vnodemarker     0     0K       -  1106958  512
      fadvise     0     0K       -        8  32
chacha20random     1     4K       -        1  4096
          BPF     3     1K       -        3  128
        ifnet     4     7K       -        4  128,2048
       ifaddr    69    23K       -       69  16,32,64,128,256,512,2048,4096
  ether_multi    45     4K       -       45  16,32,64,128
        clone     8     1K       -        8  128
      lltable    33    14K       -      134  256,512
          tun     3     1K       -        3  32
        iflib    29   206K       -       33  16,64,128,1024,2048,8192,16384,32768
         pfil     4     1K       -        4  128
     routetbl    65     9K       -       67  32,64,128,512
         vnet     1     1K       -        1  64
    vnet_data     1   240K       -        1  
vnet_data_free     1     1K       -        1  32
         igmp     3     1K       -        3  128
     in_multi     2     1K       -        2  256
  ip_moptions    17     2K       -       17  64
encap_export_host     8     1K       -        8  32,64
    sctp_a_it     0     0K       -       16  16
    sctp_atcl     0     0K       -        3  512
    sctp_atky     0     0K       -        3  32
    sctp_athm     0     0K       -        3  16
     sctp_vrf     1     1K       -        1  64
     sctp_ifa    22     3K       -       22  128
     sctp_ifn     2     1K       -        2  128
    sctp_iter     0     0K       -       16  256
   sctp_socko     0     0K       -        3  16
   tfo_ccache     1   128K       -        1  
    hostcache     1    32K       -        1  32768
          LRO     4    80K       -        4  8192,32768
      tcpfunc     1     1K       -        1  64
     syncache     1    68K       -        1  
    in6_multi    31     4K       -       31  32,256
 ip6_moptions     3     1K       -        3  32
       ip6opt    10     3K       -       13  256
          mld     3     1K       -        3  128
       ip6ndp     6     1K       -        8  64,256
  inpcbpolicy    76     3K       -    21959  32
     secasvar     1     1K       -        1  1024
       sahead     1     1K       -        1  1024
  ipsecpolicy     1     1K       -        1  1024
    ipsec-saq     2     2K       -        2  1024
       crypto     2     3K       -        2  1024,2048
          rpc    19    14K       -     1847  64,128,512,1024,4096
audit_evclass   230     8K       -      286  32
    ufs_quota     1  8192K       -        1  
    vm_pgdata     1     1K       -        1  128
      UMAHash    48 77832K       -      223  512,1024,2048,4096,8192,16384,32768,65536
      memdesc     1     4K       -        1  4096
     pci_link    16     2K       -       16  32,128
     atkbddev     2     1K       -        2  64
      acpisem    56     7K       -       56  128
      acpidev    81     6K       -       81  64
      CAM SIM    11     3K       -       11  256
       apmdev     1     1K       -        1  128
   madt_table     0     0K       -        2  128,4096
         intr     4   388K       -        4  65536
      io_apic     2     6K       -        2  2048,4096
   local_apic     1    16K       -        1  16384
          MCA    40    11K       -       40  128,512
         cpus     2     1K       -        2  64
          msi    29     4K       -       29  64,128
     nexusdev     5     1K       -        5  16
      solaris 2477753 964172K       - 4159470550  16,32,64,128,256,512,1024,2048,4096,8192,16384,32768
        linux    31     2K       -      620  64
   kstat_data     7     1K       -        7  64
    sfs_nodes   120    60K       -      256  512
  mirror_data     3     1K       -     1189  64,256,512
  fdesc_mount     1     1K       -       69  16
  nullfs_hash     1  8192K       -        1  
  nullfs_node   802    51K       -   182702  64
 nullfs_mount    11     1K       -     3750  32
  tmpfs mount     0     0K       -     3278  256
   tmpfs name     0     0K       - 188451304  16,32,64,128,256
     iprtheap 55096 95308K       -  1119121  32,64,128,256,512,1024,2048,4096,8192,16384,32768,65536
     netgraph     2     1K       -        2  64
 netgraph_msg     0     0K       -        2  1024
netgraph_hook     4     1K       -        4  128
netgraph_node     5     1K       -        5  128,256

------------------------------------------------------------------------
vmstat -z

ITEM                   SIZE  LIMIT     USED     FREE      REQ     FAILSLEEP XDOMAIN

UMA Kegs:               216,      0,     204,      11,    6760,   0,   0,   0
UMA Zones:             2472,      0,     212,       0,    6768,   0,   0,   0
UMA Slabs:              112,      0, 4189165,  397189,21997896,   0,   0,   0
UMA Hash:               256,      0,      53,      18,     101,   0,   0,   0
4 Bucket:                32,      0,   19052,   20779, 5957830,  30,   0,   0
6 Bucket:                48,      0,    2992,   15586, 1940385,3157,   0,   0
8 Bucket:                64,      0,    2039,   14388, 1502924,   0,   0,   0
12 Bucket:               96,      0,    2185,    8658, 1025798,   0,   0,   0
16 Bucket:              128,      0,   11076,    4164, 1931591,   0,   0,   0
32 Bucket:              256,      0,    7574,    4117, 2722836,4709,   0,   0
64 Bucket:              512,      0,   16530,   13442, 6084103,217744,   0,   0
128 Bucket:            1024,      0,    5352,    2718, 2032603,739180,   0,   0
256 Bucket:            2048,      0,   19723,    9306,92172361,2480863,   0,   0
vmem:                  1856,      0,       0,       1,       0,   0,   0,   0
vmem btag:               56,      0,  204679,  159193,  811411,2496,   0,   0
VM OBJECT:              256,      0,  260550, 2926625,462561319,   0,   0,   0
RADIX NODE:             144,      0, 2047368,  856097,1582729750,   0,   0,   0
MAP:                    232,      0,       2,      65,       2,   0,   0,   0
KMAP ENTRY:             112,      0,     413,    1615,     413,   0,   0,   0
MAP ENTRY:              112,      0,   21796,    9187,1200861481,   0,   0,   0
VMSPACE:               2552,      0,      96,     837,27817091,   0,   0,   0
fakepg:                 104,      0,       0,       0,       0,   0,   0,   0
64 pcpu:                  8,      0,    3870,    3953,   23937,   0,   0,   0
mt_stats_zone:           64,      0,     449,     255,     449,   0,   0,   0
mt_zone:                 24,      0,     449,     547,     449,   0,   0,   0
16:                      16,      0,      59,    2929,     210,   0,   0,   0
16:                      16,      0,    2897,    4075,   50547,   0,   0,   0
16:                      16,      0,      15,    2973,     155,   0,   0,   0
16:                      16,      0,  110446,   67266,483375391,   0,   0,   0
16:                      16,      0,     635,   17050,  491596,   0,   0,   0
16:                      16,      0,    5922,  513913,58252165,   0,   0,   0
16:                      16,      0,      21,    4212,    7139,   0,   0,   0
16:                      16,      0,      14,    4469,  476333,   0,   0,   0
32:                      32,      0,   12953,    2175,  265455,   0,   0,   0
32:                      32,      0,    3495,    4069,   28303,   0,   0,   0
32:                      32,      0,     100,    4116,   21983,   0,   0,   0
32:                      32,      0,  332604,  195628,1089651921,   0,   0,   0
32:                      32,      0,     187,    4153,  122831,   0,   0,   0
32:                      32,      0,    3825,  381080,86598620,   0,   0,   0
32:                      32,      0,     271,    4317, 2558735,   0,   0,   0
32:                      32,      0,     183,    5164, 2197314,   0,   0,   0
64:                      64,      0,     180,    3974, 4870424,   0,   0,   0
64:                      64,      0,     740,    5444,29187515,   0,   0,   0
64:                      64,      0,      19,     291,      19,   0,   0,   0
64:                      64,      0,  247332, 2969910,186788021,   0,   0,   0
64:                      64,      0,     432,   14825,  356628,   0,   0,   0
64:                      64,      0,    9057,  162806,70762526,   0,   0,   0
64:                      64,      0,    1145,   24615, 2653064,   0,   0,   0
64:                      64,      0,      52,    5342,321329175,   0,   0,   0
128:                    128,      0,   26680,     871,14603024,   0,   0,   0
128:                    128,      0,    2191,    1498,    3047,   0,   0,   0
128:                    128,      0,      37,     242,      37,   0,   0,   0
128:                    128,      0,   21722,   35433,410507086,   0,   0,   0
128:                    128,      0,     244,    2020,    4509,   0,   0,   0
128:                    128,      0,    2601,   16061,  652857,   0,   0,   0
128:                    128,      0,     863,    9315,10572787,   0,   0,   0
128:                    128,      0,     537,    2667, 6778226,   0,   0,   0
256:                    256,      0,      26,     979, 5814150,   0,   0,   0
256:                    256,      0,      39,     981,  147046,   0,   0,   0
256:                    256,      0,      10,     590,     146,   0,   0,   0
256:                    256,      0, 1634737,    8199,309907594,   0,   0,   0
256:                    256,      0,     356,    1803, 4457210,   0,   0,   0
256:                    256,      0,    3036,    4602,30386887,   0,   0,   0
256:                    256,      0,      14,     976,    6780,   0,   0,   0
256:                    256,      0,       3,    1167, 2381908,   0,   0,   0
512:                    512,      0,   11125,     222,14121609,   0,   0,   0
512:                    512,      0,      49,     455,   62768,   0,   0,   0
512:                    512,      0,       1,       6,       1,   0,   0,   0
512:                    512,      0,   84101,   13899,483103566,   0,   0,   0
512:                    512,      0,      14,       7,      24,   0,   0,   0
512:                    512,      0,     399,     561, 1116209,   0,   0,   0
512:                    512,      0,      25,     465,    6739,   0,   0,   0
512:                    512,      0,       1,     412,    1444,   0,   0,   0
1024:                  1024,      0,      11,     251,58917325,   0,   0,   0
1024:                  1024,      0,      19,      13,      23,   0,   0,   0
1024:                  1024,      0,       6,      14,       6,   0,   0,   0
1024:                  1024,      0,    1733,    1543,21634854,   0,   0,   0
1024:                  1024,      0,      24,     208,  140149,   0,   0,   0
1024:                  1024,      0,     211,     209,    1983,   0,   0,   0
1024:                  1024,      0,       3,       9,       4,   0,   0,   0
1024:                  1024,      0,       0,      16,      35,   0,   0,   0
2048:                  2048,      0,      15,      69,    1243,   0,   0,   0
2048:                  2048,      0,      35,      95,    7543,   0,   0,   0
2048:                  2048,      0,     300,     578,37030622,   0,   0,   0
2048:                  2048,      0,      94,     109,1149308183,   0,   0,   0
2048:                  2048,      0,       7,       1,       9,   0,   0,   0
2048:                  2048,      0,      11,       5,     161,   0,   0,   0
2048:                  2048,      0,      16,       4,      56,   0,   0,   0
2048:                  2048,      0,       0,       0,       0,   0,   0,   0
4096:                  4096,      0,      11,      26,23786972,   0,   0,   0
4096:                  4096,      0,      14,      18,   56619,   0,   0,   0
4096:                  4096,      0,     357,      30,     667,   0,   0,   0
4096:                  4096,      0,   75614,     200,60262103,   0,   0,   0
4096:                  4096,      0,       6,      45,23975537,   0,   0,   0
4096:                  4096,      0,      96,       0,   14538,   0,   0,   0
4096:                  4096,      0,      13,       0,      30,   0,   0,   0
4096:                  4096,      0,       0,       3,      34,   0,   0,   0
8192:                  8192,      0,   10743,       0,   10901,   0,   0,   0
8192:                  8192,      0,      10,       0,      10,   0,   0,   0
8192:                  8192,      0,      17,       0,      17,   0,   0,   0
8192:                  8192,      0,      67,      64,   48967,   0,   0,   0
8192:                  8192,      0,       2,       0,       2,   0,   0,   0
8192:                  8192,      0,      16,       0,     523,   0,   0,   0
8192:                  8192,      0,       1,       9,     254,   0,   0,   0
8192:                  8192,      0,       0,       0,       0,   0,   0,   0
16384:                16384,      0,       3,       4,     194,   0,   0,   0
16384:                16384,      0,       3,       0,       3,   0,   0,   0
16384:                16384,      0,       2,       0,       2,   0,   0,   0
16384:                16384,      0,       2,      27,   56978,   0,   0,   0
16384:                16384,      0,       0,       0,       0,   0,   0,   0
16384:                16384,      0,       2,       0,     781,   0,   0,   0
16384:                16384,      0,       0,       0,      94,   0,   0,   0
16384:                16384,      0,       0,       1,       1,   0,   0,   0
32768:                32768,      0,       6,      15,    4860,   0,   0,   0
32768:                32768,      0,       7,       0,       7,   0,   0,   0
32768:                32768,      0,       4,       0,       4,   0,   0,   0
32768:                32768,      0,      13,       4,      36,   0,   0,   0
32768:                32768,      0,       0,       0,       0,   0,   0,   0
32768:                32768,      0,       0,       0,     676,   0,   0,   0
32768:                32768,      0,       0,       3,       7,   0,   0,   0
32768:                32768,      0,       1,       2,       6,   0,   0,   0
65536:                65536,      0,       5,      18,    1585,   0,   0,   0
65536:                65536,      0,       2,       0,       2,   0,   0,   0
65536:                65536,      0,       2,       0,       2,   0,   0,   0
65536:                65536,      0,       2,       0,       2,   0,   0,   0
65536:                65536,      0,       1,       0,      33,   0,   0,   0
65536:                65536,      0,      14,       0,     608,   0,   0,   0
65536:                65536,      0,       0,       0,       4,   0,   0,   0
65536:                65536,      0,       0,       0,       0,   0,   0,   0
SLEEPQUEUE:              88,      0,    2143,     678,    2143,   0,   0,   0
Files:                   80,      0,     606,    4965,1560292384,   0,   0,   0
filedesc0:             1104,      0,     187,     330,27775793,   0,   0,   0
rl_entry:                40,      0,    1548,    2412,    1548,   0,   0,   0
rangeset pctrie nodes:    144,      0,       0,       0,       0,   0,   0,   0
TURNSTILE:              136,      0,    2143,     497,    2143,   0,   0,   0
umtx pi:                 96,      0,       0,       0,       0,   0,   0,   0
umtx_shm:                88,      0,       0,     270,     132,   0,   0,   0
swrbuf:                 832,    128,    1582,    1013, 3004574,   0,   0,   0
swwbuf:                 832,     64,      50,     981, 5983348,   0,   0,   0
nfspbuf:                832,    128,       0,     970,       0,   0,   0,   0
mdpbuf:                 832,     32,       0,     970,       0,   0,   0,   0
clpbuf:                 832,    128,       0,     970,       0,   0,   0,   0
vnpbuf:                 832,    512,       0,     970,       0,   0,   0,   0
pbuf:                   832,     32,       0,     970,       0,   0,   0,   0
MAC labels:              40,      0,       0,       0,       0,   0,   0,   0
PROC:                  1328,      0,     164,     847,27775770,   0,   0,   0
THREAD:                1440,      0,    1592,     579, 1261961,   0,   0,   0
cpuset:                 104,      0,      45,    2063,  280073,   0,   0,   0
domainset:               40,      0,       0,    4092,  279892,   0,   0,   0
audit_record:          1280,      0,       0,       0,       0,   0,   0,   0
mbuf_packet:            256,      0,       5,    6850,  406991,   0,   0,   0
mbuf:                   256, 26129792,    2146,    7295, 7193590,   0,   0,   0
mbuf_cluster:          2048, 4082780,   47845,    1275,  736532,   0,   0,   0
mbuf_jumbo_page:       4096, 2041390,       5,    1126, 1015376,   0,   0,   0
mbuf_jumbo_9k:         9216, 604856,       0,       0,       0,   0,   0,   0
mbuf_jumbo_16k:       16384, 340231,       0,       0,       0,   0,   0,   0
mbuf_extpgs:            256,      0,       0,       0,       0,   0,   0,   0
epoch_record pcpu:      256,      0,       4,      12,       4,   0,   0,   0
FPU_save_area:          832,      0,       0,       0,       0,   0,   0,   0
ttyinq:                 160,      0,     180,     156,     390,   0,   0,   0
ttyoutq:                256,      0,      96,      99,     208,   0,   0,   0
DMAR_MAP_ENTRY:         120,      0,       0,       0,       0,   0,   0,   0
nvme_request:           128,      0,       0,       0,       0,   0,   0,   0
g_bio:                  384,      0,    2310,    3250,116868679,   0,   0,   0
cryptop:                128,      0,       0,       0,       0,   0,   0,   0
cryptodesc:             120,      0,       0,       0,       0,   0,   0,   0
crypto_session:          24,      0,       0,       0,       0,   0,   0,   0
vtnet_tx_hdr:            24,      0,       0,       0,       0,   0,   0,   0
taskq_zone:              48,      0,       0,    5312, 4454846,   0,   0,   0
VNODE:                  480,      0, 1164895,    5190,185275583,   0,   0,   0
VNODEPOLL:              120,      0,      12,     120,      13,   0,   0,   0
BUF TRIE:               144,      0,       3,  105945,       3,   0,   0,   0
NAMEI:                 1024,      0,      47,     584,4365207274,   0,   0,   0
rentr:                   24,      0,       0,    4150,  434815,   0,   0,   0
S VFS Cache:            108,      0,  876780,  379790,422323092,   0,   0,   0
STS VFS Cache:          148,      0,     104,     806,     472,   0,   0,   0
L VFS Cache:            328,      0,  360073,  251021,63456485,   0,   0,   0
LTS VFS Cache:          368,      0,       6,      84,      13,   0,   0,   0
DIRHASH:               1024,      0,       0,       0,       0,   0,   0,   0
NCLNODE:                592,      0,     123,     129,     187,   0,   0,   0
AIO:                    208,      0,       0,       0,       0,   0,   0,   0
AIOP:                    32,      0,       0,       0,       0,   0,   0,   0
AIOCB:                  752,      0,       0,       0,       0,   0,   0,   0
AIOLIO:                 280,      0,       0,       0,       0,   0,   0,   0
pipe:                   760,      0,      30,     295,13851948,   0,   0,   0
procdesc:               136,      0,       0,     928,    1295,   0,   0,   0
reference_cache:         40,      0,      24,    5223,32828610,   0,   0,   0
reference_history_cache:      8,      0,      24,    4209,32828610,   0,   0,   0
range_seg_cache:         72,      0,  828817,  723998,31887063,   0,   0,   0
zio_cache:             1048,      0,      20,    9058,611289523,   0,   0,   0
zio_link_cache:          48,      0,       0,   12782,234340536,   0,   0,   0
zio_buf_512:            512,      0, 2183709, 3137075,107264614,   0,   0,   0
zio_data_buf_512:       512,      0,    5387,     801,17094118,   0,   0,   0
zio_buf_1024:          1024,      0,  114610,    1115, 1068201,   0,   0,   0
zio_data_buf_1024:     1024,      0,    9528,    1320,  567767,   0,   0,   0
zio_buf_1536:          1536,      0,   52187,     311,  347085,   0,   0,   0
zio_data_buf_1536:     1536,      0,    6690,     360,  142406,   0,   0,   0
zio_buf_2048:          2048,      0,   23870,     170, 1049170,   0,   0,   0
zio_data_buf_2048:     2048,      0,    4438,     232,  152151,   0,   0,   0
zio_buf_2560:          2560,      0,   15009,      63,   47621,   0,   0,   0
zio_data_buf_2560:     2560,      0,    3685,      57,  103107,   0,   0,   0
zio_buf_3072:          3072,      0,   15044,      63,   56325,   0,   0,   0
zio_data_buf_3072:     3072,      0,    3145,      49,   72357,   0,   0,   0
zio_buf_3584:          3584,      0,    8876,      63,  297458,   0,   0,   0
zio_data_buf_3584:     3584,      0,    2574,      47,   37734,   0,   0,   0
zio_buf_4096:          4096,      0,  755786,    5311,10326573,   0,   0,   0
zio_data_buf_4096:     4096,      0,   16214,     399, 2290240,   0,   0,   0
zio_buf_5120:          5120,      0,       0,      30,   56040,   0,   0,   0
zio_data_buf_5120:     5120,      0,       9,      32,  489652,   0,   0,   0
zio_buf_6144:          6144,      0,       0,      29,   23054,   0,   0,   0
zio_data_buf_6144:     6144,      0,       0,      32,   52418,   0,   0,   0
zio_buf_7168:          7168,      0,       0,      25,   35027,   0,   0,   0
zio_data_buf_7168:     7168,      0,       3,      32,   37350,   0,   0,   0
zio_buf_8192:          8192,      0,      12,      63, 2088389,   0,   0,   0
zio_data_buf_8192:     8192,      0,       0,     128,   75306,   0,   0,   0
zio_buf_10240:        10240,      0,       0,      68,   61496,   0,   0,   0
zio_data_buf_10240:   10240,      0,       2,      32,  507701,   0,   0,   0
zio_buf_12288:        12288,      0,      15,      92, 1360200,   0,   0,   0
zio_data_buf_12288:   12288,      0,       0,      78,  105386,   0,   0,   0
zio_buf_14336:        14336,      0,       8,      27,   19120,   0,   0,   0
zio_data_buf_14336:   14336,      0,       0,      32,  342345,   0,   0,   0
zio_buf_16384:        16384,      0,   80717,    1133,24156015,   0,   0,   0
zio_data_buf_16384:   16384,      0,       1,     126,   97425,   0,   0,   0
zio_buf_20480:        20480,      0,       0,      85,  511608,   0,   0,   0
zio_data_buf_20480:   20480,      0,       0,      74,  376130,   0,   0,   0
zio_buf_24576:        24576,      0,       4,      62,  724473,   0,   0,   0
zio_data_buf_24576:   24576,      0,       4,      32,  367142,   0,   0,   0
zio_buf_28672:        28672,      0,       3,      50,  326452,   0,   0,   0
zio_data_buf_28672:   28672,      0,       2,      32,  344129,   0,   0,   0
zio_buf_32768:        32768,      0,       0,      56,  455726,   0,   0,   0
zio_data_buf_32768:   32768,      0,       4,      78,  349709,   0,   0,   0
zio_buf_40960:        40960,      0,       0,      54,  461326,   0,   0,   0
zio_data_buf_40960:   40960,      0,       8,      70,  701100,   0,   0,   0
zio_buf_49152:        49152,      0,       0,      51,  356571,   0,   0,   0
zio_data_buf_49152:   49152,      0,       1,      32,  632357,   0,   0,   0
zio_buf_57344:        57344,      0,       0,      56,  248131,   0,   0,   0
zio_data_buf_57344:   57344,      0,       0,      80,  632598,   0,   0,   0
zio_buf_65536:        65536,      0,       0,      54,  181623,   0,   0,   0
zio_data_buf_65536:   65536,      0,       0,      32,  592992,   0,   0,   0
zio_buf_81920:        81920,      0,       3,      52,  216624,   0,   0,   0
zio_data_buf_81920:   81920,      0,       0,      32, 1147831,   0,   0,   0
zio_buf_98304:        98304,      0,       4,      54,  132444,   0,   0,   0
zio_data_buf_98304:   98304,      0,       4,      32, 1103587,   0,   0,   0
zio_buf_114688:      114688,      0,       0,      23,   65614,   0,   0,   0
zio_data_buf_114688: 114688,      0,       0,      30, 1089740,   0,   0,   0
zio_buf_131072:      131072,      0,     365,    2345, 1289946,   0,   0,   0
zio_data_buf_131072: 131072,      0,   14170,    2099, 6731576,   0,   0,   0
zio_buf_163840:      163840,      0,       0,      25,   34236,   0,   0,   0
zio_data_buf_163840: 163840,      0,       0,       0,       0,   0,   0,   0
zio_buf_196608:      196608,      0,       0,      23,   14794,   0,   0,   0
zio_data_buf_196608: 196608,      0,       0,       0,       0,   0,   0,   0
zio_buf_229376:      229376,      0,       0,      21,    7068,   0,   0,   0
zio_data_buf_229376: 229376,      0,       0,       0,       0,   0,   0,   0
zio_buf_262144:      262144,      0,       2,      19,  471392,   0,   0,   0
zio_data_buf_262144: 262144,      0,       0,       0,       0,   0,   0,   0
zio_buf_327680:      327680,      0,       1,      20,    6619,   0,   0,   0
zio_data_buf_327680: 327680,      0,       0,       0,       0,   0,   0,   0
zio_buf_393216:      393216,      0,       0,      20,  173677,   0,   0,   0
zio_data_buf_393216: 393216,      0,       0,       0,       0,   0,   0,   0
zio_buf_458752:      458752,      0,       0,      20,    2327,   0,   0,   0
zio_data_buf_458752: 458752,      0,       0,       0,       0,   0,   0,   0
zio_buf_524288:      524288,      0,       0,      20,   75401,   0,   0,   0
zio_data_buf_524288: 524288,      0,       0,       0,       0,   0,   0,   0
zio_buf_655360:      655360,      0,       0,      19,   37810,   0,   0,   0
zio_data_buf_655360: 655360,      0,       0,       0,       0,   0,   0,   0
zio_buf_786432:      786432,      0,       0,      19,   20323,   0,   0,   0
zio_data_buf_786432: 786432,      0,       0,       0,       0,   0,   0,   0
zio_buf_917504:      917504,      0,       0,      19,   11815,   0,   0,   0
zio_data_buf_917504: 917504,      0,       0,       0,       0,   0,   0,   0
zio_buf_1048576:     1048576,      0,       0,      20,   22909,   0,   0,   0
zio_data_buf_1048576: 1048576,      0,       0,       0,       0,   0,   0,   0
zio_buf_1310720:     1310720,      0,       0,       0,       0,   0,   0,   0
zio_data_buf_1310720: 1310720,      0,       0,       0,       0,   0,   0,   0
zio_buf_1572864:     1572864,      0,       0,       0,       0,   0,   0,   0
zio_data_buf_1572864: 1572864,      0,       0,       0,       0,   0,   0,   0
zio_buf_1835008:     1835008,      0,       0,       0,       0,   0,   0,   0
zio_data_buf_1835008: 1835008,      0,       0,       0,       0,   0,   0,   0
zio_buf_2097152:     2097152,      0,       0,       0,       0,   0,   0,   0
zio_data_buf_2097152: 2097152,      0,       0,       0,       0,   0,   0,   0
zio_buf_2621440:     2621440,      0,       0,       0,       0,   0,   0,   0
zio_data_buf_2621440: 2621440,      0,       0,       0,       0,   0,   0,   0
zio_buf_3145728:     3145728,      0,       0,       0,       0,   0,   0,   0
zio_data_buf_3145728: 3145728,      0,       0,       0,       0,   0,   0,   0
zio_buf_3670016:     3670016,      0,       0,       0,       0,   0,   0,   0
zio_data_buf_3670016: 3670016,      0,       0,       0,       0,   0,   0,   0
zio_buf_4194304:     4194304,      0,       0,       0,       0,   0,   0,   0
zio_data_buf_4194304: 4194304,      0,       0,       0,       0,   0,   0,   0
zio_buf_5242880:     5242880,      0,       0,       0,       0,   0,   0,   0
zio_data_buf_5242880: 5242880,      0,       0,       0,       0,   0,   0,   0
zio_buf_6291456:     6291456,      0,       0,       0,       0,   0,   0,   0
zio_data_buf_6291456: 6291456,      0,       0,       0,       0,   0,   0,   0
zio_buf_7340032:     7340032,      0,       0,       0,       0,   0,   0,   0
zio_data_buf_7340032: 7340032,      0,       0,       0,       0,   0,   0,   0
zio_buf_8388608:     8388608,      0,       0,       0,       0,   0,   0,   0
zio_data_buf_8388608: 8388608,      0,       0,       0,       0,   0,   0,   0
zio_buf_10485760:    10485760,      0,       0,       0,       0,   0,   0,   0
zio_data_buf_10485760: 10485760,      0,       0,       0,       0,   0,   0,   0
zio_buf_12582912:    12582912,      0,       0,       0,       0,   0,   0,   0
zio_data_buf_12582912: 12582912,      0,       0,       0,       0,   0,   0,   0
zio_buf_14680064:    14680064,      0,       0,       0,       0,   0,   0,   0
zio_data_buf_14680064: 14680064,      0,       0,       0,       0,   0,   0,   0
zio_buf_16777216:    16777216,      0,       0,       0,       0,   0,   0,   0
zio_data_buf_16777216: 16777216,      0,       0,       0,       0,   0,   0,   0
lz4_ctx:              16384,      0,       0,     130, 9312170,   0,   0,   0
abd_chunk:             4096,      0, 3162229,   13164,368244656,   0,   0,   0
sa_cache:               152,      0, 1170686,    1553,76826252,   0,   0,   0
dnode_t:                952,      0, 2001487, 2441125,76865423,   0,   0,   0
arc_buf_hdr_t_full:     376,      0, 1538816,     326,68468244,   0,   0,   0
arc_buf_hdr_t_l2only:     96,      0,       0,       0,       0,   0,   0,   0
arc_buf_t:               64,      0,   96053,  215802,60142127,   0,   0,   0
dmu_buf_impl_t:         352,      0, 2095715, 3502178,126695158,   0,   0,   0
zil_lwb_cache:          320,      0,       7,     499,     781,   0,   0,   0
zil_zcw_cache:           80,      0,       1,    1764,     953,   0,   0,   0
sio_cache:              128,      0,       0,       0,       0,   0,   0,   0
zfs_znode_cache:        272,      0, 1170105,    1201,74416925,   0,   0,   0
Mountpoints:           2744,      0,      87,     376,    7444,   0,   0,   0
ksiginfo:               112,      0,     931,    4785,  266889,   0,   0,   0
itimer:                 352,      0,       2,     175,   10739,   0,   0,   0
KNOTE:                  160,      0,      28,    1364, 1701528,   0,   0,   0
socket:                 872, 2094056,     144,    3904,  711315,   0,   0,   0
IPsec SA lft_c:          16,      0,       0,       0,       0,   0,   0,   0
unpcb:                  256, 2094056,      67,    1163,  689344,   0,   0,   0
ipq:                     56,  51200,       0,       0,       0,   0,   0,   0
udp_inpcb:              488, 2094056,      37,     731,   21027,   0,   0,   0
udpcb:                   32, 2094056,      37,    4055,   21027,   0,   0,   0
tcp_inpcb:              488, 2094056,      39,     433,     930,   0,   0,   0
tcpcb:                  976, 2094056,      39,     205,     930,   0,   0,   0
tcptw:                   88,  27767,       0,    1035,      47,   0,   0,   0
syncache:               168,  15360,       0,     736,      37,   0,   0,   0
hostcache:               96,  15360,       1,     409,      24,   0,   0,   0
sackhole:                32,      0,       0,       0,       0,   0,   0,   0
tfo:                      4,      0,       0,       0,       0,   0,   0,   0
tfo_ccache_entries:      80,      0,       0,       0,       0,   0,   0,   0
tcpreass:                48, 255173,       0,     581,     359,   0,   0,   0
tcp_log:                408, 5000000,       0,       0,       0,   0,   0,   0
tcp_log_bucket:         144,      0,       0,       0,       0,   0,   0,   0
tcp_log_node:           120,      0,       0,       0,       0,   0,   0,   0
sctp_ep:               1280, 2094056,       0,       9,       3,   0,   0,   0
sctp_asoc:             2408,  40000,       0,       0,       0,   0,   0,   0
sctp_laddr:              48,  80000,       0,       0,      21,   0,   0,   0
sctp_raddr:             736,  80000,       0,       0,       0,   0,   0,   0
sctp_chunk:             152, 400000,       0,       0,       0,   0,   0,   0
sctp_readq:             152, 400000,       0,       0,       0,   0,   0,   0
sctp_stream_msg_out:    112, 400000,       0,       0,       0,   0,   0,   0
sctp_asconf:             40, 400000,       0,       0,       0,   0,   0,   0
sctp_asconf_ack:         48, 400000,       0,       0,       0,   0,   0,   0
udplite_inpcb:          488, 2094056,       0,       0,       0,   0,   0,   0
ripcb:                  488, 2094056,       0,      64,       2,   0,   0,   0
rtentry:                208,      0,      48,     180,      50,   0,   0,   0
selfd:                   64,      0,    2149,    4602,62171728,   0,   0,   0
swpctrie:               144, 8165610,   10172,   64359,  250717,   0,   0,   0
swblk:                  136, 8165588,   65694, 1923029, 6648015,   0,   0,   0
NetGraph items:          72,   4096,       0,     155,       4,   0,   0,   0
NetGraph data items:     72,   4096,       0,    2015, 1063619,   0,   0,   0
Received on Wed Sep 25 2019 - 20:52:12 UTC

This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:41:22 UTC