[CRASH] ZFS recv

From: Larry Rosenman <ler_at_lerctr.org>
Date: Mon, 25 Mar 2013 09:01:30 -0500 (CDT)
Greetings,
     I'm getting a zfs recv crash on 10.0-CURRENT....

I was ssh'd to my 8.4 box and did:
zfs send -R -D vault_at_2013-03-24 | ssh home.lerctr.org zfs recv -F -u -v -d zroot/backups/TBH

And on the 2nd (or so) filesystem got the following panic.

I have the vmcore as well as sources, and can give ssh access.  I can also 
reproduce this at will.

ideas?
borg.lerctr.org dumped core - see /var/crash/vmcore.4

Mon Mar 25 08:52:46 CDT 2013

FreeBSD borg.lerctr.org 10.0-CURRENT FreeBSD 10.0-CURRENT #129 r248695: Mon Mar 25 05:03:32 CDT 2013     root_at_borg.lerctr.org:/usr/obj/usr/src/sys/BORG-DTRACE  amd64

panic: page fault

GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd"...

Unread portion of the kernel message buffer:


Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 00
fault virtual address	= 0x378
fault code		= supervisor read data, page not present
instruction pointer	= 0x20:0xffffffff80531426
stack pointer	        = 0x28:0xffffff91579193d0
frame pointer	        = 0x28:0xffffff9157919470
code segment		= base 0x0, limit 0xfffff, type 0x1b
 			= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags	= interrupt enabled, resume, IOPL = 0
current process		= 1044 (zfs)
trap number		= 12
panic: page fault
cpuid = 0
Uptime: 2m10s
Dumping 4913 out of 64747 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91%

Reading symbols from /boot/kernel/zfs.ko...Reading symbols from /boot/kernel/zfs.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/zfs.ko
Reading symbols from /boot/kernel/acl_nfs4.ko...Reading symbols from /boot/kernel/acl_nfs4.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/acl_nfs4.ko
Reading symbols from /boot/kernel/opensolaris.ko...Reading symbols from /boot/kernel/opensolaris.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/opensolaris.ko
Reading symbols from /boot/kernel/linux.ko...Reading symbols from /boot/kernel/linux.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/linux.ko
Reading symbols from /boot/kernel/coretemp.ko...Reading symbols from /boot/kernel/coretemp.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/coretemp.ko
Reading symbols from /boot/kernel/ichsmb.ko...Reading symbols from /boot/kernel/ichsmb.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/ichsmb.ko
Reading symbols from /boot/kernel/smbus.ko...Reading symbols from /boot/kernel/smbus.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/smbus.ko
Reading symbols from /boot/kernel/ichwd.ko...Reading symbols from /boot/kernel/ichwd.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/ichwd.ko
Reading symbols from /boot/kernel/cpuctl.ko...Reading symbols from /boot/kernel/cpuctl.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/cpuctl.ko
Reading symbols from /boot/kernel/crypto.ko...Reading symbols from /boot/kernel/crypto.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/crypto.ko
Reading symbols from /boot/kernel/cryptodev.ko...Reading symbols from /boot/kernel/cryptodev.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/cryptodev.ko
Reading symbols from /boot/kernel/dtraceall.ko...Reading symbols from /boot/kernel/dtraceall.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/dtraceall.ko
Reading symbols from /boot/kernel/profile.ko...Reading symbols from /boot/kernel/profile.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/profile.ko
Reading symbols from /boot/kernel/cyclic.ko...Reading symbols from /boot/kernel/cyclic.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/cyclic.ko
Reading symbols from /boot/kernel/dtrace.ko...Reading symbols from /boot/kernel/dtrace.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/dtrace.ko
Reading symbols from /boot/kernel/systrace_freebsd32.ko...Reading symbols from /boot/kernel/systrace_freebsd32.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/systrace_freebsd32.ko
Reading symbols from /boot/kernel/systrace.ko...Reading symbols from /boot/kernel/systrace.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/systrace.ko
Reading symbols from /boot/kernel/sdt.ko...Reading symbols from /boot/kernel/sdt.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/sdt.ko
Reading symbols from /boot/kernel/lockstat.ko...Reading symbols from /boot/kernel/lockstat.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/lockstat.ko
Reading symbols from /boot/kernel/fasttrap.ko...Reading symbols from /boot/kernel/fasttrap.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/fasttrap.ko
Reading symbols from /boot/kernel/fbt.ko...Reading symbols from /boot/kernel/fbt.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/fbt.ko
Reading symbols from /boot/kernel/dtnfscl.ko...Reading symbols from /boot/kernel/dtnfscl.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/dtnfscl.ko
Reading symbols from /boot/kernel/dtmalloc.ko...Reading symbols from /boot/kernel/dtmalloc.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/dtmalloc.ko
Reading symbols from /boot/kernel/dtio.ko...Reading symbols from /boot/kernel/dtio.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/dtio.ko
Reading symbols from /boot/modules/vboxdrv.ko...done.
Loaded symbols for /boot/modules/vboxdrv.ko
Reading symbols from /boot/kernel/fdescfs.ko...Reading symbols from /boot/kernel/fdescfs.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/fdescfs.ko
Reading symbols from /boot/kernel/uhid.ko...Reading symbols from /boot/kernel/uhid.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/uhid.ko
Reading symbols from /boot/modules/vboxnetflt.ko...done.
Loaded symbols for /boot/modules/vboxnetflt.ko
Reading symbols from /boot/kernel/netgraph.ko...Reading symbols from /boot/kernel/netgraph.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/netgraph.ko
Reading symbols from /boot/kernel/ng_ether.ko...Reading symbols from /boot/kernel/ng_ether.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/ng_ether.ko
Reading symbols from /boot/modules/vboxnetadp.ko...done.
Loaded symbols for /boot/modules/vboxnetadp.ko
#0  doadump (textdump=<value optimized out>) at pcpu.h:229
229	pcpu.h: No such file or directory.
 	in pcpu.h
(kgdb) #0  doadump (textdump=<value optimized out>) at pcpu.h:229
#1  0xffffffff80529060 in kern_reboot (howto=260)
     at /usr/src/sys/kern/kern_shutdown.c:447
#2  0xffffffff805293e7 in panic (fmt=<value optimized out>)
     at /usr/src/sys/kern/kern_shutdown.c:754
#3  0xffffffff80732cd5 in trap_fatal (frame=<value optimized out>,
     eva=<value optimized out>) at /usr/src/sys/amd64/amd64/trap.c:872
#4  0xffffffff807330d9 in trap_pfault (frame=0x0, usermode=0)
     at /usr/src/sys/amd64/amd64/trap.c:730
#5  0xffffffff807326bc in trap (frame=0xffffff9157919320)
     at /usr/src/sys/amd64/amd64/trap.c:463
#6  0xffffffff8071cc62 in calltrap () at exception.S:228
#7  0xffffffff80531426 in _sx_xlock_hard (sx=0xfffffe01d5b64db8,
     tid=18446741877920023696, opts=<value optimized out>, file=0x0,
     line=250541281) at /usr/src/sys/kern/kern_sx.c:556
#8  0xffffffff80530f8c in _sx_xlock (sx=0x0, opts=0, file=0x0, line=0)
     at sx.h:152
#9  0xffffffff80e67251 in dmu_objset_from_ds (ds=0xfffffe01d5b64c00,
     osp=0xffffff91579195b0)
     at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_objset.c:434
#10 0xffffffff80e649ef in dmu_recv_stream (drc=0xffffff9157919750,
     fp=<value optimized out>, voffp=0xffffff9157919740, cleanup_fd=8,
     action_handlep=<value optimized out>)
     at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_send.c:1318
#11 0xffffffff80ed9226 in zfs_ioc_recv (zc=0xffffff8012cc8000)
     at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_ioctl.c:4084
#12 0xffffffff80ed44a1 in zfsdev_ioctl (dev=<value optimized out>,
     zcmd=<value optimized out>,
     arg=0xffffff8012cc8000 "zroot/backups/TBH/home",
     flag=<value optimized out>, td=<value optimized out>)
     at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_ioctl.c:5902
#13 0xffffffff80427e0f in devfs_ioctl_f (fp=0xfffffe00c0b360a0,
     com=3517471259, data=0xffffff8012cc8000, cred=<value optimized out>,
     td=0xfffffe00c0bec490) at /usr/src/sys/fs/devfs/devfs_vnops.c:757
#14 0xffffffff8057359b in kern_ioctl (td=0xfffffe00c0bec490,
     fd=<value optimized out>, com=18446741877920023696) at file.h:306
#15 0xffffffff8057331f in sys_ioctl (td=0xfffffe00c0bec490,
     uap=0xffffff9157919b80) at /usr/src/sys/kern/sys_generic.c:693
#16 0xffffffff807335d7 in amd64_syscall (td=0xfffffe00c0bec490, traced=0)
     at subr_syscall.c:134
#17 0xffffffff8071cf4b in Xfast_syscall () at exception.S:387
#18 0x00000008019dacea in ?? ()
Previous frame inner to this frame (corrupt stack?)
Current language:  auto; currently minimal
(kgdb)

------------------------------------------------------------------------
ps -axl

  UID  PID PPID CPU PRI NI    VSZ  RSS MWCHAN   STAT TT     TIME COMMAND
    0    0    0   0  -8  0      0    0 -        DLs   -  0:02.51 [kernel]
    0    1    0   0  20  0   9428    0 wait     DLs   -  0:00.10 [init]
    0    2    0   0 -16  0      0    0 crypto_w DL    -  0:00.00 [crypto]
    0    3    0   0 -16  0      0    0 crypto_r DL    -  0:00.00 [crypto returns
    0    4    0   0 -16  0      0    0 -        DL    -  0:00.00 [fdc0]
    0    5    0   0  -8  0      0    0 tx->tx_s DL    -  0:00.08 [zfskern]
    0    6    0   0 -16  0      0    0 waiting_ DL    -  0:00.00 [sctp_iterator]
    0    7    0   0 -16  0      0    0 ccb_scan DL    -  0:00.00 [xpt_thrd]
    0    8    0   0 -16  0      0    0 -        RL    -  0:00.00 [pagedaemon]
    0    9    0   0 -16  0      0    0 psleep   DL    -  0:00.00 [vmdaemon]
    0   10    0   0 -16  0      0    0 audit_wo DL    -  0:00.00 [audit]
    0   11    0   0 155  0      0    0 -        RL    -  6:41.28 [idle]
    0   12    0   0 -84  0      0    0 -        WL    -  0:00.26 [intr]
    0   13    0   0  -8  0      0    0 -        DL    -  0:00.33 [geom]
    0   14    0   0 -16  0      0    0 -        DL    -  0:00.01 [yarrow]
    0   15    0   0 -68  0      0    0 -        DL    -  0:00.01 [usb]
    0   16    0   0 -20  0      0    0 VBoxIS   DL    -  0:00.00 [TIMER]
    0   17    0   0 155  0      0    0 pgzero   DL    -  0:00.00 [pagezero]
    0   18    0   0 -16  0      0    0 psleep   DL    -  0:00.00 [bufdaemon]
    0   19    0   0  16  0      0    0 -        RL    -  0:00.00 [syncer]
    0   20    0   0 -16  0      0    0 vlruwt   DL    -  0:00.00 [vnlru]
    0  632    1   0  20  0  13196    0 select   Ds    -  0:00.00 [devd]
    0  757    1   0  20  0  14376    0 select   Ds    -  0:00.04 [syslogd]
    0  760    1   0 -52  0   6180 2088 -        Rs    -  0:00.00 [watchdogd]
    0  774    1   0  20  0  16456    0 select   Ds    -  0:00.01 [rpcbind]
    0  809    1   0  52  0  38948    0 select   Ds    -  0:00.02 [mountd]
    0  815    1   0  52  0  36792    0 select   Ds    -  0:00.04 [nfsd]
    0  817  815   0  52  0  12216    0 rpcsvc   D     -  0:00.00 [nfsd]
    0  853    1   0  20  0  25196    0 select   Ds    -  0:00.01 [ntpd]
    0  874    0   0 -16  0      0    0 sleep    DL    -  0:00.00 [ng_queue]
    0  886    1   0  20  0  34708    0 nanslp   Ds    -  0:00.01 [perl]
    0  890    1   0  52  0  30780    0 nanslp   D     -  0:00.43 [smartd]
   70  899    1   0  20  0  84416    0 select   Ds    -  0:00.22 [postgres]
   70  903  899   0  20  0  84416    0 select   Ds    -  0:00.00 [postgres]
   70  904  899   0  20  0  84416    0 select   Ds    -  0:00.00 [postgres]
   70  905  899   0  20  0  84416    0 select   Ds    -  0:00.00 [postgres]
   70  906  899   0  20  0  44120    0 select   Ds    -  0:00.00 [postgres]
   26  928    1   0  52  0  48644    0 select   Ds    -  0:00.00 [exim-4.80.1-2]
    0  938    1   0  20  0 103216    0 -        Rs    -  0:00.02 [cupsd]
1028  944    1   0 155  0  40308    0 select   Ds    -  0:00.05 [boinc_client]
  910  948    1   0  20  0  59412    0 uwait    Ds    -  0:00.00 [bacula-sd]
    0  951    1   0  20  0  57068    0 uwait    Ds    -  0:00.00 [bacula-fd]
  910  954    1   0  20  0  72404    0 -        Rs    -  0:00.00 [bacula-dir]
    0  959    1   0  20  0  56264    0 select   Ds    -  0:00.00 [sshd]
    0  969    1   0  20  0  16460    0 nanslp   Ds    -  0:00.00 [cron]
    0  990    1   0  52  0  18576    0 select   Ds    -  0:00.00 [inetd]
    0 1010    1   0  21  0  47584    0 wait     Ds    -  0:00.00 [login]
    0 1011    1   0  52  0  14364    0 ttyin    Ds+   -  0:00.00 [getty]
    0 1012    1   0  52  0  14364    0 ttyin    Ds+   -  0:00.00 [getty]
    0 1013    1   0  52  0  14364    0 ttyin    Ds+   -  0:00.00 [getty]
    0 1014    1   0  52  0  14364    0 ttyin    Ds+   -  0:00.00 [getty]
    0 1015    1   0  52  0  14364    0 ttyin    Ds+   -  0:00.00 [getty]
    0 1016    1   0  52  0  14364    0 ttyin    Ds+   -  0:00.00 [getty]
    0 1017    1   0  52  0  14364    0 ttyin    Ds+   -  0:00.00 [getty]
    0 1020 1010   0  20  0  16928    0 wait     D     -  0:00.00 [sh]
    0 1022 1020   0  20  0  47640    0 select   D+    -  0:00.00 [ssh]
1028 1024  944   0 155 19  54272    0 -        RN    -  0:17.51 [wcg_hpf2_roset
1028 1025  944   0 155 19  54272    0 -        RN    -  0:18.11 [wcg_hpf2_roset
1028 1026  944   0 155 19 129292    0 -        RN    -  0:03.57 [wcgrid_faah_7.
1028 1027  944   0 155 19   1892    0 -        RN    -  0:00.03 [wcgrid_cep2_6.
1028 1028  944   0 155 19  74216    0 m        DN    -  0:17.23 [setiathome-6.1
1028 1029  944   0 155 19  74216    0 -        RN    -  0:19.34 [setiathome-6.1
1028 1030  944   0 155 19  74216    0 m        DN    -  0:16.21 [setiathome-6.1
1028 1031  944   0 155 19 129292    0 i        DN    -  0:00.00 [wcgrid_faah_7.
1028 1032  944   0 155 19  74216    0 m        DN    -  0:17.25 [setiathome-6.1
1028 1033 1027   0 155 19   1892    0 i        DN    -  0:00.00 [wcgrid_cep2_6.
1028 1034 1033   0 155 19   1892    0 i        DN    -  0:00.00 [wcgrid_cep2_6.
1028 1035 1024   0 155 19  54272    0 6        DN    -  0:00.00 [wcg_hpf2_roset
1028 1036 1035   0 155 19  54272    0 -        RN    -  0:00.00 [wcg_hpf2_roset
1028 1037 1025   0 155 19  54272    0 6        DN    -  0:00.00 [wcg_hpf2_roset
1028 1038 1037   0 155 19  54272    0 6        DN    -  0:00.00 [wcg_hpf2_roset
    0 1040  959   0  20  0  81532    0 select   Ds    -  0:00.00 [sshd]
    0 1042 1040   0  27  0  20304    0 pause    Ds    -  0:00.00 [csh]
    0 1044 1042   0  21  0  39968    0 -        R     -  0:00.00 [zfs]

------------------------------------------------------------------------
vmstat -s

    339379 cpu context switches
     36990 device interrupts
      6821 software interrupts
    362408 traps
  34071033 system calls
        21 kernel threads created
       834  fork() calls
       182 vfork() calls
         7 rfork() calls
         0 swap pager pageins
         0 swap pager pages paged in
         0 swap pager pageouts
         0 swap pager pages paged out
      1567 vnode pager pageins
     11054 vnode pager pages paged in
         8 vnode pager pageouts
        16 vnode pager pages paged out
         0 page daemon wakeups
         0 pages examined by the page daemon
        11 pages reactivated
     33160 copy-on-write faults
       209 copy-on-write optimized faults
    284971 zero fill pages zeroed
         0 zero fill pages prezeroed
       107 intransit blocking page faults
    405648 total VM faults taken
      3427 page faults requiring I/O
         0 pages affected by kernel thread creation
     32298 pages affected by  fork()
      6408 pages affected by vfork()
     22897 pages affected by rfork()
         0 pages cached
    407800 pages freed
         0 pages freed by daemon
         0 pages freed by exiting processes
     67444 pages active
     20205 pages inactive
         8 pages in VM cache
    860785 pages wired down
  15199607 pages free
      4096 bytes per page
    247996 total name lookups
           cache hits (93% pos + 1% neg) system 0% per-directory
           deletions 0%, falsehits 0%, toolong 0%

------------------------------------------------------------------------
vmstat -m

          Type InUse MemUse HighUse Requests  Size(s)
        DEVFS3   207    52K       -      226  256
      filedesc    78   251K       -     1058  16,32,64,128,2048,4096
filedesc_to_leader    11     1K       -       11  64
         sigio     1     1K       -        1  64
       kdtrace   500   112K       -     1633  64,256
          kenv    84    11K       -      112  16,32,64,128
        kqueue     2     3K       -       52  256,2048
     proc-args    46     4K       -      373  16,32,64,128,256
        DEVFS1   181    91K       -      199  512
         hhook     2     1K       -        2  128
       ithread   119    21K       -      119  32,128,256
        KTRACE   100    13K       -      100  128
         DEVFS    40     1K       -       41  16,128
        linker   351   171K       -      477  16,32,64,128,256,512,1024,2048,4096
        DEVFSP     1     1K       -       11  64
         lockf    50     6K       -       86  64,128
    loginclass     2     1K       -        6  64
         cache     1     1K       -        1  32
        devbuf 17037 34759K       -    17243  16,32,64,128,256,512,1024,2048,4096
          temp    56    35K       -    10350  16,32,64,128,256,512,1024,2048,4096
        ip6ndp     5     1K       -        6  64,128
        module   293    37K       -      293  128
      mtx_pool     2    16K       -        2
           osd     8     1K       -       20  16,32,64,128
      pmchooks     1     1K       -        1  128
          pgrp    36     5K       -       51  128
       session    34     5K       -       45  128
          proc     2   256K       -        2
       subproc   231   350K       -     1202  512,4096
          cred   108    17K       -     8456  64,256
        plimit    19     5K       -      202  256
       uidinfo     6    33K       -        9  128
        NFS fh     1     1K       -        7  16
        sysctl     0     0K       -      380  16,32,64
     sysctloid  6406   317K       -     6553  16,32,64,128
     sysctltmp     0     0K       -      338  16,32,64,128,256,4096
       tidhash     1   256K       -        1
       callout     9  3208K       -        9
          umtx  1062   133K       -     1062  128
      p1003.1b     1     1K       -        1  16
          SWAP    12 19681K       -       12  64
           bus   900    86K       -     4751  16,32,64,128,256,512,1024
        bus-sc   135   290K       -     1989  16,32,64,128,256,512,1024,2048,4096
       devstat    24    49K       -       24  32,4096
  eventhandler    86     7K       -       86  64,128
          kobj   169   676K       -      681  4096
       Per-cpu     1     1K       -        1  32
          rman   292    33K       -      686  16,32,128
          sbuf     0     0K       -     2785  16,32,64,128,256,512,1024,2048,4096
         stack     0     0K       -        2  256
     taskqueue   105    16K       -      135  16,32,64,256,1024
        Unitno    19     2K       -      227  32,64
      ioctlops     1     8K       -    13701  16,32,64,128,256,512,1024,2048
        select    34     5K       -       34  128
           iov     0     0K       -     1301  16,64,128,256,512
           msg     4    30K       -        4  2048,4096
           sem     4   106K       -        4  2048,4096
           shm    11    40K       -       22  2048
           tty    19    19K       -       21  1024,2048
      mbuf_tag     0     0K       -       58  32,128
         shmfd     1     8K       -        1
        soname     7     1K       -      697  16,32,128
           pcb    38  8341K       -       95  16,32,128,1024,2048
      vfscache     1 16384K       -        1
      vfs_hash     1  8192K       -        1
        vnodes     1     1K       -        1  256
         mount   397    20K       -     1897  16,32,64,128,256,512
   vnodemarker     0     0K       -      202  512
           BPF     3     1K       -        3  128
         ifnet     4     7K       -        4  128,2048
        ifaddr    48    15K       -       48  32,64,128,256,512,2048,4096
   ether_multi    40     3K       -       46  16,32,64
         clone     6     1K       -        6  128
        arpcom     2     1K       -        2  16
       lltable    12     5K       -       12  256,512
     newnfsmnt     1     1K       -        1  1024
     pfs_nodes    21     6K       -       21  256
   pfs_vncache    76     5K       -       81  64
          GEOM   344    60K       -     2226  16,32,64,128,256,512,1024,2048
      routetbl    35     5K       -      240  32,64,128,256,512
          igmp     3     1K       -        3  256
      in_multi     2     1K       -        2  256
      ppbusdev     2     1K       -        2  256
       entropy  1024    64K       -     1024  64
     sctp_a_it     0     0K       -        3  16
      sctp_vrf     1     1K       -        1  64
      sctp_ifa     5     1K       -        5  128
      sctp_ifn     2     1K       -        2  128
     sctp_iter     0     0K       -        3  256
     hostcache     1    28K       -        1
      syncache     1   100K       -        1
   in6_mfilter     1     1K       -        1  1024
     in6_multi    22     3K       -       22  32,256
  ip6_moptions     2     1K       -        2  32,256
       CAM DEV    15    30K       -       24  2048
          UART     6     5K       -        6  16,1024
           mld     3     1K       -        3  128
       ata_pci     1     1K       -        1  64
       CAM CCB    13    26K       -      101  2048
           rpc    22     6K       -       28  32,64,128,256,512,1024
audit_evclass   188     6K       -      227  32
     vm_pgdata     7  8193K       -        7  128
       UMAHash     1     1K       -        1  512
     raid_data     0     0K       -      336  32,128,256
      acpiintr     1     1K       -        1  64
        acpica  1830   187K       -    57239  16,32,64,128,256,512,1024,2048
      CAM path    22     1K       -       68  32
    CAM periph    16     4K       -       41  16,32,64,128,256
      acpitask     1     8K       -        1
       acpisem    21     3K       -       21  128
     CAM queue    39     2K       -      133  16,32,64
       memdesc     1     4K       -        1  4096
       acpidev    36     3K       -       36  64
      atkbddev     2     1K       -        2  64
CAM dev queue     8     1K       -        8  128
           USB    38    41K       -       40  16,32,64,128,256,512,1024,4096
md_nvidia_data     0     0K       -       54  512
        USBdev    30     6K       -       40  64,128,512
   md_sii_data     0     0K       -       54  512
       CAM SIM     8     2K       -        8  256
       CAM XPT    58     4K       -      204  16,32,64,128,1024
        kbdmux     7    18K       -        7  16,512,1024,2048
        apmdev     1     1K       -        1  128
    madt_table     0     0K       -        1  4096
           LED     4     1K       -        4  16,128
        isadev     5     1K       -        5  128
       io_apic     2     4K       -        2  2048
           MCA     8     1K       -        8  128
      pci_link    16     2K       -       16  32,64,128
           msi     2     1K       -        2  128
      nexusdev     4     1K       -        4  16
     acpi_perf     8     1K       -        8  64
       scsi_cd     0     0K       -       11  16
          cdev     8     2K       -        8  256
       solaris 62374 344297K       -   749641  16,32,64,128,256,512,1024,2048,4096
         linux    36     2K       -       36  32,64
        cpuctl     1     1K       -        9  64,4096
        crypto     1     1K       -        1  512
         xform     0     0K       -      206  16,32
        cyclic    32     3K       -       32  16,64,128
    kstat_data     5     1K       -        5  64
           fbt     1   256K       -        1
      iprtheap    23    56K       -       23  32,64,128,256,2048
   fdesc_mount     1     1K       -        1  16
netgraph_node     4     1K       -        4  128,256
      netgraph     2     1K       -        2  64

------------------------------------------------------------------------
vmstat -z

ITEM                   SIZE  LIMIT     USED     FREE      REQ FAIL SLEEP

UMA Kegs:               208,      0,     254,       1,     254,   0,   0
UMA Zones:             1408,      0,     254,       0,     254,   0,   0
UMA Slabs:              568,      0,   16533,     120,   31690,   0,   0
UMA RCntSlabs:          568,      0,    1217,       1,    1217,   0,   0
UMA Hash:               256,      0,      82,       8,      83,   0,   0
16 Bucket:              152,      0,     391,       9,     391,   0,   0
32 Bucket:              280,      0,     317,       5,     317,  15,   0
64 Bucket:              536,      0,     358,       6,     358,  84,   0
128 Bucket:            1048,      0,    1365,       0,    1365,4185,   0
VM OBJECT:              240,      0,    3359,     257,   14724,   0,   0
RADIX NODE:             144, 16148184,   20410,16127826,   76441,   0,   0
MAP:                    232,      0,       8,      24,       8,   0,   0
KMAP ENTRY:             120, 4230725,     255,    1233,   48042,   0,   0
MAP ENTRY:              120,      0,    1670,     407,   34695,   0,   0
fakepg:                 104,      0,       0,       0,       0,   0,   0
mt_zone:               4112,      0,     283,      22,     283,   0,   0
16:                      16,      0,       6,    1170,    8417,   0,   0
16:                      16,      0,    3274,     758,   56874,   0,   0
16:                      16,      0,       2,     166,       2,   0,   0
16:                      16,      0,      87,    1089,   23969,   0,   0
16:                      16,      0,    2339,    1021,    2911,   0,   0
16:                      16,      0,     550,     962,    3590,   0,   0
16:                      16,      0,       8,     832,      41,   0,   0
16:                      16,      0,      62,     946,     270,   0,   0
32:                      32,      0,      31,     878,    4358,   0,   0
32:                      32,      0,    2152,    1989,   31736,   0,   0
32:                      32,      0,      29,     577,      29,   0,   0
32:                      32,      0,     122,     787,    4245,   0,   0
32:                      32,      0,    2388,     945,    2665,   0,   0
32:                      32,      0,     241,     971,    2379,   0,   0
32:                      32,      0,      71,     838,     280,   0,   0
32:                      32,      0,     200,     406,     329,   0,   0
64:                      64,      0,      84,     476,     810,   0,   0
64:                      64,      0,   24198,    3130,  222936,   0,   0
64:                      64,      0,    1107,     517,    2080,   0,   0
64:                      64,      0,     729,     559,   25092,   0,   0
64:                      64,      0,     324,     460,     358,   0,   0
64:                      64,      0,    8983,     873,   16224,   0,   0
64:                      64,      0,      33,     135,      33,   0,   0
64:                      64,      0,      65,     495,     237,   0,   0
128:                    128,      0,      34,     198,      80,   0,   0
128:                    128,      0,    9778,    1300,   81357,   0,   0
128:                    128,      0,      85,     234,     100,   0,   0
128:                    128,      0,    1220,     404,    2193,   0,   0
128:                    128,      0,    2953,     585,    3025,   0,   0
128:                    128,      0,     567,     361,    1209,   0,   0
128:                    128,      0,      16,     100,      16,   0,   0
128:                    128,      0,      80,     239,     427,   0,   0
256:                    256,      0,      10,      20,      12,   0,   0
256:                    256,      0,    2880,    3525,  105923,   0,   0
256:                    256,      0,     490,      80,     783,   0,   0
256:                    256,      0,     240,      75,    1045,   0,   0
256:                    256,      0,       5,      70,       7,   0,   0
256:                    256,      0,     302,     208,    4700,   0,   0
256:                    256,      0,      22,      23,      22,   0,   0
256:                    256,      0,     129,     216,    1507,   0,   0
512:                    512,      0,     169,      41,     210,   0,   0
512:                    512,      0,   10146,     347,  128131,   0,   0
512:                    512,      0,       4,      59,      76,   0,   0
512:                    512,      0,     217,      77,    1180,   0,   0
512:                    512,      0,      12,      44,     120,   0,   0
512:                    512,      0,     137,     136,     357,   0,   0
512:                    512,      0,       0,       0,       0,   0,   0
512:                    512,      0,       0,     252,     877,   0,   0
1024:                  1024,      0,       2,      66,      84,   0,   0
1024:                  1024,      0,     325,     187,    2938,   0,   0
1024:                  1024,      0,       4,       4,       4,   0,   0
1024:                  1024,      0,       6,      14,    1917,   0,   0
1024:                  1024,      0,      19,       9,      19,   0,   0
1024:                  1024,      0,      15,     321,    2674,   0,   0
1024:                  1024,      0,       1,       7,       1,   0,   0
1024:                  1024,      0,      13,      15,     164,   0,   0
2048:                  2048,      0,      38,      96,     326,   0,   0
2048:                  2048,      0,     411,     237,    4477,   0,   0
2048:                  2048,      0,       0,       0,       0,   0,   0
2048:                  2048,      0,      81,     133,    1080,   0,   0
2048:                  2048,      0,       1,       7,       3,   0,   0
2048:                  2048,      0,      23,      95,     170,   0,   0
2048:                  2048,      0,       0,       0,       0,   0,   0
2048:                  2048,      0,       0,       8,      65,   0,   0
4096:                  4096,      0,      67,     100,    1038,   0,   0
4096:                  4096,      0,    6394,    4781,   99018,   0,   0
4096:                  4096,      0,     169,     100,     681,   0,   0
4096:                  4096,      0,      24,      41,      28,   0,   0
4096:                  4096,      0,      10,      44,      12,   0,   0
4096:                  4096,      0,      27,      63,    7787,   0,   0
4096:                  4096,      0,       0,       0,       0,   0,   0
4096:                  4096,      0,       0,       0,       0,   0,   0
Files:                   80,      0,     192,     393,   17327,   0,   0
TURNSTILE:              136,      0,     532,     108,     532,   0,   0
rl_entry:                40,      0,      75,     681,      75,   0,   0
umtx pi:                 96,      0,       0,       0,       0,   0,   0
MAC labels:              40,      0,       0,       0,       0,   0,   0
PROC:                  1208,      0,      73,      89,    1044,   0,   0
THREAD:                1168,      0,     425,     106,     587,   0,   0
SLEEPQUEUE:              80,      0,     532,     164,     532,   0,   0
VMSPACE:                392,      0,      46,     134,    1022,   0,   0
cpuset:                  72,      0,     274,     326,     425,   0,   0
cyclic_id_cache:         64,      0,       0,       0,       0,   0,   0
audit_record:          1240,      0,       0,       0,       0,   0,   0
mbuf_packet:            256, 26520435,    1023,    1163,    1760,   0,   0
mbuf:                   256, 26520435,       7,    1027,    1874,   0,   0
mbuf_cluster:          2048, 4143818,    2176,     230,    2176,   0,   0
mbuf_jumbo_page:       4096, 2071909,       0,      14,       7,   0,   0
mbuf_jumbo_9k:         9216, 1841694,       0,       0,       0,   0,   0
mbuf_jumbo_16k:       16384, 1381272,       0,       0,       0,   0,   0
mbuf_ext_refcnt:          4,      0,       0,       0,       0,   0,   0
dtrace_state_cache:    4096,      0,       0,       0,       0,   0,   0
g_bio:                  248,      0,       2,     538,  109938,   0,   0
ttyinq:                 160,      0,     120,     144,     255,   0,   0
ttyoutq:                256,      0,      64,     101,     136,   0,   0
ata_request:            336,      0,       0,     143,      77,   0,   0
ata_composite:          336,      0,       0,       0,       0,   0,   0
vtnet_tx_hdr:            24,      0,       0,       0,       0,   0,   0
cryptop:                 88,      0,       0,       0,       0,   0,   0
cryptodesc:              72,      0,       0,       0,       0,   0,   0
FPU_save_area:          512,      0,       0,       0,       0,   0,   0
taskq_zone:              48,      0,       0,     504,      92,   0,   0
VNODE:                  472,      0,    9393,     279,    9518,   0,   0
VNODEPOLL:              112,      0,       0,       0,       0,   0,   0
NAMEI:                 1024,      0,       0,     128,   53101,   0,   0
S VFS Cache:            108,      0,    9441,     360,   11402,   0,   0
STS VFS Cache:          148,      0,       0,       0,       0,   0,   0
L VFS Cache:            328,      0,      84,     132,     112,   0,   0
LTS VFS Cache:          368,      0,       0,       0,       0,   0,   0
NCLNODE:                528,      0,       1,      13,       1,   0,   0
pipe:                   744,      0,      12,      93,     553,   0,   0
space_seg_cache:         64,      0,    9593,   80007,  278530,   0,   0
zio_cache:              944,      0,       5,    7883,  318981,   0,   0
zio_link_cache:          48,      0,       3,    8709,  303395,   0,   0
zio_buf_512:            512,      0,       0,       0,       0,   0,   0
zio_data_buf_512:       512,      0,       0,       0,       0,   0,   0
zio_buf_1024:          1024,      0,       0,       0,       0,   0,   0
zio_data_buf_1024:     1024,      0,       0,       0,       0,   0,   0
zio_buf_1536:          1536,      0,       0,       0,       0,   0,   0
zio_data_buf_1536:     1536,      0,       0,       0,       0,   0,   0
zio_buf_2048:          2048,      0,       0,       0,       0,   0,   0
zio_data_buf_2048:     2048,      0,       0,       0,       0,   0,   0
zio_buf_2560:          2560,      0,       0,       0,       0,   0,   0
zio_data_buf_2560:     2560,      0,       0,       0,       0,   0,   0
zio_buf_3072:          3072,      0,       0,       0,       0,   0,   0
zio_data_buf_3072:     3072,      0,       0,       0,       0,   0,   0
zio_buf_3584:          3584,      0,       0,       0,       0,   0,   0
zio_data_buf_3584:     3584,      0,       0,       0,       0,   0,   0
zio_buf_4096:          4096,      0,       0,       0,       0,   0,   0
zio_data_buf_4096:     4096,      0,       0,       0,       0,   0,   0
zio_buf_5120:          5120,      0,       0,       0,       0,   0,   0
zio_data_buf_5120:     5120,      0,       0,       0,       0,   0,   0
zio_buf_6144:          6144,      0,       0,       0,       0,   0,   0
zio_data_buf_6144:     6144,      0,       0,       0,       0,   0,   0
zio_buf_7168:          7168,      0,       0,       0,       0,   0,   0
zio_data_buf_7168:     7168,      0,       0,       0,       0,   0,   0
zio_buf_8192:          8192,      0,       0,       0,       0,   0,   0
zio_data_buf_8192:     8192,      0,       0,       0,       0,   0,   0
zio_buf_10240:        10240,      0,       0,       0,       0,   0,   0
zio_data_buf_10240:   10240,      0,       0,       0,       0,   0,   0
zio_buf_12288:        12288,      0,       0,       0,       0,   0,   0
zio_data_buf_12288:   12288,      0,       0,       0,       0,   0,   0
zio_buf_14336:        14336,      0,       0,       0,       0,   0,   0
zio_data_buf_14336:   14336,      0,       0,       0,       0,   0,   0
zio_buf_16384:        16384,      0,       0,       0,       0,   0,   0
zio_data_buf_16384:   16384,      0,       0,       0,       0,   0,   0
zio_buf_20480:        20480,      0,       0,       0,       0,   0,   0
zio_data_buf_20480:   20480,      0,       0,       0,       0,   0,   0
zio_buf_24576:        24576,      0,       0,       0,       0,   0,   0
zio_data_buf_24576:   24576,      0,       0,       0,       0,   0,   0
zio_buf_28672:        28672,      0,       0,       0,       0,   0,   0
zio_data_buf_28672:   28672,      0,       0,       0,       0,   0,   0
zio_buf_32768:        32768,      0,       0,       0,       0,   0,   0
zio_data_buf_32768:   32768,      0,       0,       0,       0,   0,   0
zio_buf_36864:        36864,      0,       0,       0,       0,   0,   0
zio_data_buf_36864:   36864,      0,       0,       0,       0,   0,   0
zio_buf_40960:        40960,      0,       0,       0,       0,   0,   0
zio_data_buf_40960:   40960,      0,       0,       0,       0,   0,   0
zio_buf_45056:        45056,      0,       0,       0,       0,   0,   0
zio_data_buf_45056:   45056,      0,       0,       0,       0,   0,   0
zio_buf_49152:        49152,      0,       0,       0,       0,   0,   0
zio_data_buf_49152:   49152,      0,       0,       0,       0,   0,   0
zio_buf_53248:        53248,      0,       0,       0,       0,   0,   0
zio_data_buf_53248:   53248,      0,       0,       0,       0,   0,   0
zio_buf_57344:        57344,      0,       0,       0,       0,   0,   0
zio_data_buf_57344:   57344,      0,       0,       0,       0,   0,   0
zio_buf_61440:        61440,      0,       0,       0,       0,   0,   0
zio_data_buf_61440:   61440,      0,       0,       0,       0,   0,   0
zio_buf_65536:        65536,      0,       0,       0,       0,   0,   0
zio_data_buf_65536:   65536,      0,       0,       0,       0,   0,   0
zio_buf_69632:        69632,      0,       0,       0,       0,   0,   0
zio_data_buf_69632:   69632,      0,       0,       0,       0,   0,   0
zio_buf_73728:        73728,      0,       0,       0,       0,   0,   0
zio_data_buf_73728:   73728,      0,       0,       0,       0,   0,   0
zio_buf_77824:        77824,      0,       0,       0,       0,   0,   0
zio_data_buf_77824:   77824,      0,       0,       0,       0,   0,   0
zio_buf_81920:        81920,      0,       0,       0,       0,   0,   0
zio_data_buf_81920:   81920,      0,       0,       0,       0,   0,   0
zio_buf_86016:        86016,      0,       0,       0,       0,   0,   0
zio_data_buf_86016:   86016,      0,       0,       0,       0,   0,   0
zio_buf_90112:        90112,      0,       0,       0,       0,   0,   0
zio_data_buf_90112:   90112,      0,       0,       0,       0,   0,   0
zio_buf_94208:        94208,      0,       0,       0,       0,   0,   0
zio_data_buf_94208:   94208,      0,       0,       0,       0,   0,   0
zio_buf_98304:        98304,      0,       0,       0,       0,   0,   0
zio_data_buf_98304:   98304,      0,       0,       0,       0,   0,   0
zio_buf_102400:      102400,      0,       0,       0,       0,   0,   0
zio_data_buf_102400: 102400,      0,       0,       0,       0,   0,   0
zio_buf_106496:      106496,      0,       0,       0,       0,   0,   0
zio_data_buf_106496: 106496,      0,       0,       0,       0,   0,   0
zio_buf_110592:      110592,      0,       0,       0,       0,   0,   0
zio_data_buf_110592: 110592,      0,       0,       0,       0,   0,   0
zio_buf_114688:      114688,      0,       0,       0,       0,   0,   0
zio_data_buf_114688: 114688,      0,       0,       0,       0,   0,   0
zio_buf_118784:      118784,      0,       0,       0,       0,   0,   0
zio_data_buf_118784: 118784,      0,       0,       0,       0,   0,   0
zio_buf_122880:      122880,      0,       0,       0,       0,   0,   0
zio_data_buf_122880: 122880,      0,       0,       0,       0,   0,   0
zio_buf_126976:      126976,      0,       0,       0,       0,   0,   0
zio_data_buf_126976: 126976,      0,       0,       0,       0,   0,   0
zio_buf_131072:      131072,      0,       0,       0,       0,   0,   0
zio_data_buf_131072: 131072,      0,       0,       0,       0,   0,   0
sa_cache:                80,      0,    9194,     211,    9286,   0,   0
dnode_t:                856,      0,    9829,     335,   11053,   0,   0
dmu_buf_impl_t:         224,      0,   17993,     384,   21797,   0,   0
arc_buf_hdr_t:          216,      0,   11850,     264,   12999,   0,   0
arc_buf_t:               72,      0,   10407,     443,   14007,   0,   0
zil_lwb_cache:          192,      0,       4,     176,      56,   0,   0
zfs_znode_cache:        368,      0,    9194,     226,    9286,   0,   0
Mountpoints:            816,      0,      30,      25,      30,   0,   0
ksiginfo:               112,      0,     313,     842,    3205,   0,   0
itimer:                 352,      0,       1,      21,       1,   0,   0
KNOTE:                  128,      0,       6,     226,      67,   0,   0
socket:                 680, 2071914,      57,      93,     480,   0,   0
unpcb:                  240, 2071920,      11,     165,     114,   0,   0
ipq:                     56, 129528,       0,       0,       0,   0,   0
udp_inpcb:              392, 2071920,      21,      99,     324,   0,   0
udpcb:                   16, 2071944,      21,    1155,     324,   0,   0
tcp_inpcb:              392, 2071920,      24,      96,      36,   0,   0
tcpcb:                 1016, 2071916,      24,      44,      36,   0,   0
tcptw:                   72,  27800,       0,     100,       2,   0,   0
syncache:               152,  15375,       0,      50,       1,   0,   0
hostcache:              136,  15372,       0,       0,       0,   0,   0
tcpreass:                40, 259056,       0,       0,       0,   0,   0
sackhole:                32,      0,       0,       0,       0,   0,   0
sctp_ep:               1408, 2071914,       0,       0,       0,   0,   0
sctp_asoc:             2344,  40000,       0,       0,       0,   0,   0
sctp_laddr:              48,  80064,       0,     288,       4,   0,   0
sctp_raddr:             728,  80000,       0,       0,       0,   0,   0
sctp_chunk:             136, 400008,       0,       0,       0,   0,   0
sctp_readq:             104, 400032,       0,       0,       0,   0,   0
sctp_stream_msg_out:    104, 400032,       0,       0,       0,   0,   0
sctp_asconf:             40, 400008,       0,       0,       0,   0,   0
sctp_asconf_ack:         48, 400032,       0,       0,       0,   0,   0
ripcb:                  392, 2071920,       0,       0,       0,   0,   0
rtentry:                200,      0,      17,     135,      17,   0,   0
selfd:                   56,      0,      83,     484,    5931,   0,   0
SWAPMETA:               288, 8074092,       0,       0,       0,   0,   0
NetGraph items:          72,   4118,       0,       0,       0,   0,   0
NetGraph data items:     72,    522,       0,       0,       0,   0,   0


------------------------------------------------------------------------
vmstat -i

interrupt                          total       rate
irq1: atkbd0                           2          0
irq6: fdc0                            22          0
irq14: ata0                          131          0
irq17: uhci0 ehci0                   371          1
irq19: uhci1 ahci0+                35224        136
cpu0:timer                         21581         83
irq256: em0                         1240          4
cpu7:timer                         18860         72
cpu1:timer                         17194         66
cpu2:timer                         22448         86
cpu3:timer                         17242         66
cpu4:timer                         21918         84
cpu6:timer                         19999         77
cpu5:timer                         20715         79
Total                             196947        760

------------------------------------------------------------------------
pstat -T

192/2071909 files
0M/147455M swap space

------------------------------------------------------------------------
pstat -s

Device          512-blocks     Used    Avail Capacity
/dev/gpt/swap0    50331392        0 50331392     0%
/dev/gpt/swap1    50331392        0 50331392     0%
/dev/gpt/swap2    50331392        0 50331392     0%
/dev/gpt/swap3    50331392        0 50331392     0%
/dev/gpt/swap4    50331392        0 50331392     0%
/dev/gpt/swap5    50331392        0 50331392     0%
Total            301988352        0 301988352     0%

------------------------------------------------------------------------
iostat

iostat: kvm_read(_tk_nin): invalid address (0x0)
iostat: disabling TTY statistics
             ada0             ada1             ada2             cpu
   KB/t tps  MB/s   KB/t tps  MB/s   KB/t tps  MB/s  us ni sy in id
  14.49  24  0.33  14.56  23  0.32  14.28  24  0.33   0 10  2  0 88

------------------------------------------------------------------------
ipcs -a

Message Queues:
T           ID          KEY MODE        OWNER    GROUP    CREATOR  CGROUP                 CBYTES                 QNUM               QBYTES        LSPID        LRPID STIME    RTIME    CTIME

Shared Memory:
T           ID          KEY MODE        OWNER    GROUP    CREATOR  CGROUP         NATTCH        SEGSZ         CPID         LPID ATIME    DTIME    CTIME 
m        65536      5432001 --rw------- pgsql    pgsql    pgsql    pgsql               4     41263104          899          899  8:44:47  8:45:50  8:44:47
m        65537     28975112 --rw-rw-rw- boinc    nobody   boinc    nobody              2         8192          944         1028  8:45:34  8:45:34  8:45:33
m        65538     28975113 --rw-rw-rw- boinc    nobody   boinc    nobody              2         8192          944         1029  8:45:34  8:45:34  8:45:33
m        65539     28975114 --rw-rw-rw- boinc    nobody   boinc    nobody              2         8192          944         1030  8:45:34  8:45:34  8:45:33
m        65540     28975115 --rw-rw-rw- boinc    nobody   boinc    nobody              2         8192          944         1032  8:45:34  8:45:34  8:45:34

Semaphores:
T           ID          KEY MODE        OWNER    GROUP    CREATOR  CGROUP          NSEMS OTIME    CTIME 
s        65536      5432001 --rw------- pgsql    pgsql    pgsql    pgsql              17  8:44:47  8:44:47
s        65537      5432002 --rw------- pgsql    pgsql    pgsql    pgsql              17  8:44:47  8:44:47
s        65538      5432003 --rw------- pgsql    pgsql    pgsql    pgsql              17  8:44:47  8:44:47
s        65539      5432004 --rw------- pgsql    pgsql    pgsql    pgsql              17  8:44:47  8:44:47
s        65540      5432005 --rw------- pgsql    pgsql    pgsql    pgsql              17  8:44:47  8:44:47
s        65541      5432006 --rw------- pgsql    pgsql    pgsql    pgsql              17  8:44:47  8:44:47
s        65542      5432007 --rw------- pgsql    pgsql    pgsql    pgsql              17  8:44:47  8:44:47


------------------------------------------------------------------------
ipcs -T

msginfo:
 	msgmax:        16384	(max characters in a message)
 	msgmni:           40	(# of message queues)
 	msgmnb:         2048	(max characters in a message queue)
 	msgtql:           40	(max # of messages in system)
 	msgssz:            8	(size of a message segment)
 	msgseg:         2048	(# of message segments in system)

shminfo:
 	shmmax:    536870912	(max shared memory segment size)
 	shmmin:            1	(min shared memory segment size)
 	shmmni:          192	(max number of shared memory identifiers)
 	shmseg:          128	(max shared memory segments per process)
 	shmall:       131072	(max amount of shared memory in pages)

seminfo:
 	semmni:           50	(# of semaphore identifiers)
 	semmns:          340	(# of semaphores in system)
 	semmnu:          150	(# of undo structures in system)
 	semmsl:          340	(max # of semaphores per id)
 	semopm:          100	(max # of operations per semop call)
 	semume:           50	(max # of undo entries per process)
 	semusz:          632	(size in bytes of undo structure)
 	semvmx:        32767	(semaphore maximum value)
 	semaem:        16384	(adjust on exit max value)


------------------------------------------------------------------------
nfsstat

Client Info:
Rpc Counts:
   Getattr   Setattr    Lookup  Readlink      Read     Write    Create    Remove
         2         0         0         0         0         0         0         0
    Rename      Link   Symlink     Mkdir     Rmdir   Readdir  RdirPlus    Access
         0         0         0         0         0         0         0         0
     Mknod    Fsstat    Fsinfo  PathConf    Commit
         0         1         1         0         0
Rpc Info:
  TimedOut   Invalid X Replies   Retries  Requests
         0         0         0         0         4
Cache Info:
Attr Hits    Misses Lkup Hits    Misses BioR Hits    Misses BioW Hits    Misses
         0         0         0         0         0         0         0         0
BioRLHits    Misses BioD Hits    Misses DirE Hits    Misses Accs Hits    Misses
         0         0         0         0         0         0         0         0

Server Info:
   Getattr   Setattr    Lookup  Readlink      Read     Write    Create    Remove
         0         0         0         0         0         0         0         0
    Rename      Link   Symlink     Mkdir     Rmdir   Readdir  RdirPlus    Access
         0         0         0         0         0         0         0         0
     Mknod    Fsstat    Fsinfo  PathConf    Commit
         0         0         0         0         0
Server Ret-Failed
                 0
Server Faults
             0
Server Cache Stats:
    Inprog      Idem  Non-idem    Misses
         0         0         0         0
Server Write Gathering:
  WriteOps  WriteRPC   Opsaved
         0         0         0

------------------------------------------------------------------------
netstat -s

tcp:
 	541 packets sent
 		224 data packets (18558 bytes)
 		0 data packets (0 bytes) retransmitted
 		0 data packets unnecessarily retransmitted
 		0 resends initiated by MTU discovery
 		308 ack-only packets (123 delayed)
 		0 URG only packets
 		0 window probe packets
 		0 window update packets
 		9 control packets
 	541 packets received
 		208 acks (for 17554 bytes)
 		3 duplicate acks
 		0 acks for unsent data
 		511 packets (432616 bytes) received in-sequence
 		0 completely duplicate packets (0 bytes)
 		0 old duplicate packets
 		0 packets with some dup. data (0 bytes duped)
 		0 out-of-order packets (0 bytes)
 		0 packets (0 bytes) of data after window
 		0 window probes
 		1 window update packet
 		0 packets received after close
 		0 discarded for bad checksums
 		0 discarded for bad header offset fields
 		0 discarded because packet too short
 		0 discarded due to memory problems
 	6 connection requests
 	1 connection accept
 	0 bad connection attempts
 	0 listen queue overflows
 	0 ignored RSTs in the windows
 	7 connections established (including accepts)
 	12 connections closed (including 0 drops)
 		0 connections updated cached RTT on close
 		0 connections updated cached RTT variance on close
 		0 connections updated cached ssthresh on close
 	0 embryonic connections dropped
 	208 segments updated rtt (of 178 attempts)
 	0 retransmit timeouts
 		0 connections dropped by rexmit timeout
 	0 persist timeouts
 		0 connections dropped by persist timeout
 	0 Connections (fin_wait_2) dropped because of timeout
 	0 keepalive timeouts
 		0 keepalive probes sent
 		0 connections dropped by keepalive
 	7 correct ACK header predictions
 	318 correct data packet header predictions
 	1 syncache entry added
 		0 retransmitted
 		0 dupsyn
 		0 dropped
 		1 completed
 		0 bucket overflow
 		0 cache overflow
 		0 reset
 		0 stale
 		0 aborted
 		0 badack
 		0 unreach
 		0 zone failures
 	1 cookie sent
 	0 cookies received
 	0 hostcache entries added
 		0 bucket overflow
 	0 SACK recovery episodes
 	0 segment rexmits in SACK recovery episodes
 	0 byte rexmits in SACK recovery episodes
 	0 SACK options (SACK blocks) received
 	0 SACK options (SACK blocks) sent
 	0 SACK scoreboard overflow
 	0 packets with ECN CE bit set
 	0 packets with ECN ECT(0) bit set
 	0 packets with ECN ECT(1) bit set
 	0 successful ECN handshakes
 	0 times ECN reduced the congestion window
udp:
 	120 datagrams received
 	0 with incomplete header
 	0 with bad data length field
 	0 with bad checksum
 	0 with no checksum
 	0 dropped due to no socket
 	26 broadcast/multicast datagrams undelivered
 	0 dropped due to full socket buffers
 	0 not for hashed pcb
 	94 delivered
 	95 datagrams output
 	0 times multicast source filter matched
ip:
 	612 total packets received
 	0 bad header checksums
 	0 with size smaller than minimum
 	0 with data size < data length
 	0 with ip length > max ip packet size
 	0 with header length < data size
 	0 with data length < header length
 	0 with bad options
 	0 with incorrect version number
 	0 fragments received
 	0 fragments dropped (dup or out of space)
 	0 fragments dropped after timeout
 	0 packets reassembled ok
 	610 packets for this host
 	2 packets for unknown/unsupported protocol
 	0 packets forwarded (0 packets fast forwarded)
 	0 packets not forwardable
 	0 packets received for unknown multicast group
 	0 redirects sent
 	585 packets sent from this host
 	0 packets sent with fabricated ip header
 	0 output packets dropped due to no bufs, etc.
 	0 output packets discarded due to no route
 	0 output datagrams fragmented
 	0 fragments created
 	0 datagrams that can't be fragmented
 	0 tunneling packets that can't find gif
 	0 datagrams with bad address in header
icmp:
 	0 calls to icmp_error
 	0 errors not generated in response to an icmp message
 	0 messages with bad code fields
 	0 messages less than the minimum length
 	0 messages with bad checksum
 	0 messages with bad length
 	0 multicast echo requests ignored
 	0 multicast timestamp requests ignored
 	0 message responses generated
 	0 invalid return addresses
 	0 no return routes
igmp:
 	2 messages received
 	0 messages received with too few bytes
 	0 messages received with wrong TTL
 	0 messages received with bad checksum
 	1 V1/V2 membership query received
 	0 V3 membership queries received
 	0 membership queries received with invalid field(s)
 	1 general query received
 	0 group queries received
 	0 group-source queries received
 	0 group-source queries dropped
 	1 membership report received
 	0 membership reports received with invalid field(s)
 	0 membership reports received for groups to which we belong
 	0 V3 reports received without Router Alert
 	0 membership reports sent
arp:
 	4 ARP requests sent
 	3 ARP replies sent
 	40 ARP requests received
 	2 ARP replies received
 	42 ARP packets received
 	1 total packet dropped due to no ARP entry
 	0 ARP entrys timed out
 	0 Duplicate IPs seen
ip6:
 	51 total packets received
 	0 with size smaller than minimum
 	0 with data size < data length
 	0 with bad options
 	0 with incorrect version number
 	0 fragments received
 	0 fragments dropped (dup or out of space)
 	0 fragments dropped after timeout
 	0 fragments that exceeded limit
 	0 packets reassembled ok
 	51 packets for this host
 	0 packets forwarded
 	0 packets not forwardable
 	0 redirects sent
 	58 packets sent from this host
 	0 packets sent with fabricated ip header
 	0 output packets dropped due to no bufs, etc.
 	9 output packets discarded due to no route
 	0 output datagrams fragmented
 	0 fragments created
 	0 datagrams that can't be fragmented
 	0 packets that violated scope rules
 	0 multicast packets which we don't join
 	Input histogram:
 		UDP: 51
 	Mbuf statistics:
 		27 one mbuf
 		24 one ext mbuf
 		0 two or more ext mbuf
 	0 packets whose headers are not contiguous
 	0 tunneling packets that can't find gif
 	0 packets discarded because of too many headers
 	0 failures of source address selection
 	Source addresses selection rule applied:
icmp6:
 	0 calls to icmp6_error
 	0 errors not generated in response to an icmp6 message
 	0 errors not generated because of rate limitation
 	Output histogram:
 		neighbor solicitation: 1
 	0 messages with bad code fields
 	0 messages < minimum length
 	0 bad checksums
 	0 messages with bad length
 	Histogram of error messages to be generated:
 		0 no route
 		0 administratively prohibited
 		0 beyond scope
 		0 address unreachable
 		0 port unreachable
 		0 packet too big
 		0 time exceed transit
 		0 time exceed reassembly
 		0 erroneous header field
 		0 unrecognized next header
 		0 unrecognized option
 		0 redirect
 		0 unknown
 	0 message responses generated
 	0 messages with too many ND options
 	0 messages with bad ND options
 	0 bad neighbor solicitation messages
 	0 bad neighbor advertisement messages
 	0 bad router solicitation messages
 	0 bad router advertisement messages
 	0 bad redirect messages
 	0 path MTU changes
rip6:
 	0 messages received
 	0 checksum calculations on inbound
 	0 messages with bad checksum
 	0 messages dropped due to no socket
 	0 multicast messages dropped due to no socket
 	0 messages dropped due to full socket buffers
 	0 delivered
 	0 datagrams output

------------------------------------------------------------------------
netstat -m

1030/2190/3220 mbufs in use (current/cache/total)
1013/1393/2406/4143818 mbuf clusters in use (current/cache/total/max)
1023/1163 mbuf+clusters out of packet secondary zone in use (current/cache)
0/14/14/2071909 4k (page size) jumbo clusters in use (current/cache/total/max)
0/0/0/1841694 9k jumbo clusters in use (current/cache/total/max)
0/0/0/1381272 16k jumbo clusters in use (current/cache/total/max)
2283K/3389K/5673K bytes allocated to network (current/cache/total)
0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters)
0/0/0 requests for mbufs delayed (mbufs/clusters/mbuf+clusters)
0/0/0 requests for jumbo clusters delayed (4k/9k/16k)
0/0/0 requests for jumbo clusters denied (4k/9k/16k)
0 requests for sfbufs denied
0 requests for sfbufs delayed
0 requests for I/O initiated by sendfile
0 calls to protocol drain routines

------------------------------------------------------------------------
netstat -id

Name    Mtu Network       Address              Ipkts Ierrs Idrop    Opkts Oerrs  Coll Drop
em0    1500 <Link#1>      00:30:48:f2:29:9c      654     0     0      591     0     0    0 
em0    1500 192.168.200.0 borg                   588     -     -      585     -     -    - 
em0    1500 fe80::230:48f fe80::230:48ff:fe        0     -     -        4     -     -    - 
em1*   1500 <Link#2>      00:30:48:f2:29:9d        0     0     0        0     0     0    0 
lo0   16384 <Link#3>                              51     0     0       51     0     0    0 
lo0   16384 localhost     ::1                     51     -     -       51     -     -    - 
lo0   16384 fe80::1%lo0   fe80::1                  0     -     -        0     -     -    - 
lo0   16384 your-net      localhost                0     -     -        0     -     -    -

------------------------------------------------------------------------
netstat -anr

Routing tables

Internet:
Destination        Gateway            Flags    Refs      Use  Netif Expire
default            192.168.200.11     UGS         0      560    em0
127.0.0.1          link#3             UH          0        0    lo0
192.168.200.0/24   link#1             U           0       25    em0
192.168.200.4      link#1             UHS         0        0    lo0

Internet6:
Destination                       Gateway                       Flags      Netif Expire
::/96                             ::1                           UGRS        lo0
::1                               link#3                        UH          lo0
::ffff:0.0.0.0/96                 ::1                           UGRS        lo0
fe80::/10                         ::1                           UGRS        lo0
fe80::%em0/64                     link#1                        U           em0
fe80::230:48ff:fef2:299c%em0      link#1                        UHS         lo0
fe80::%lo0/64                     link#3                        U           lo0
fe80::1%lo0                       link#3                        UHS         lo0
ff01::%em0/32                     fe80::230:48ff:fef2:299c%em0  U           em0
ff01::%lo0/32                     ::1                           U           lo0
ff02::/16                         ::1                           UGRS        lo0
ff02::%em0/32                     fe80::230:48ff:fef2:299c%em0  U           em0
ff02::%lo0/32                     ::1                           U           lo0

------------------------------------------------------------------------
netstat -anA

Active Internet connections (including servers)
Tcpcb            Proto Recv-Q Send-Q Local Address      Foreign Address    (state)
fffffe01c28423f8 tcp4       0   1008 192.168.200.4.22   192.147.25.65.4400 ESTABLISHED
fffffe01038a5000 tcp4       0      0 192.168.200.4.1391 198.20.8.246.80    CLOSE_WAIT
fffffe01c2272be8 tcp4       0      0 127.0.0.1.31416    *.*                LISTEN
fffffe01d5855be8 tcp4       0      0 192.168.200.4.1551 192.147.25.65.22   ESTABLISHED
fffffe01c28427f0 tcp4       0      0 *.9101             *.*                LISTEN
fffffe010370f7f0 tcp4       0      0 *.22               *.*                LISTEN
fffffe010370fbe8 tcp6       0      0 *.22               *.*                LISTEN
fffffe010381dbe8 tcp4       0      0 *.9102             *.*                LISTEN
fffffe01c2273000 tcp4       0      0 *.9103             *.*                LISTEN
fffffe0103a3bbe8 tcp4       0      0 *.631              *.*                LISTEN
fffffe0103a3c000 tcp6       0      0 *.631              *.*                LISTEN
fffffe010381a000 tcp4       0      0 127.0.0.1.587      *.*                LISTEN
fffffe010381a3f8 tcp4       0      0 127.0.0.1.25       *.*                LISTEN
fffffe010381a7f0 tcp4       0      0 192.168.200.4.587  *.*                LISTEN
fffffe010381abe8 tcp4       0      0 192.168.200.4.25   *.*                LISTEN
fffffe010370e7f0 tcp4       0      0 127.0.0.1.5432     *.*                LISTEN
fffffe010370e000 tcp6       0      0 ::1.5432           *.*                LISTEN
fffffe01038a7000 tcp6       0      0 *.2049             *.*                LISTEN
fffffe01038a73f8 tcp4       0      0 *.2049             *.*                LISTEN
fffffe010381b000 tcp4       0      0 *.843              *.*                LISTEN
fffffe010381b3f8 tcp6       0      0 *.843              *.*                LISTEN
fffffe010381d3f8 tcp4       0      0 *.111              *.*                LISTEN
fffffe010381d7f0 tcp6       0      0 *.111              *.*                LISTEN
fffffe010370e3f8 tcp4       0      0 192.168.200.4.952  192.168.200.23.204 ESTABLISHED
fffffe010313bdc8 udp4       0      0 *.69               *.* 
fffffe0103106930 udp4       0      0 *.631              *.* 
fffffe0103131ab8 udp6       0      0 ::1.47400          ::1.47400 
fffffe0103199930 udp4       0      0 127.0.0.1.123      *.* 
fffffe010319a188 udp6       0      0 fe80:3::1.123      *.* 
fffffe010319a620 udp6       0      0 ::1.123            *.* 
fffffe010319a310 udp6       0      0 fe80:1::230:48ff.1 *.* 
fffffe010319a000 udp4       0      0 192.168.200.4.123  *.* 
fffffe0103199dc8 udp6       0      0 *.123              *.* 
fffffe0103199ab8 udp4       0      0 *.123              *.* 
fffffe0103199000 udp6       0      0 *.2049             *.* 
fffffe0103199498 udp4       0      0 *.2049             *.* 
fffffe0103197930 udp4       0      0 *.843              *.* 
fffffe0103197620 udp6       0      0 *.843              *.* 
fffffe0103199c40 udp6       0      0 *.*                *.* 
fffffe010314cab8 udp4       0      0 *.1009             *.* 
fffffe010314c7a8 udp4       0      0 *.111              *.* 
fffffe010314c498 udp6       0      0 *.998              *.* 
fffffe010314c188 udp6       0      0 *.111              *.* 
fffffe0103197310 udp4       0      0 *.514              *.* 
fffffe0103197498 udp6       0      0 *.514              *.* 
Active UNIX domain sockets
Address  Type   Recv-Q Send-Q    Inode     Conn     Refs  Nextref Addr
fffffe0103a5b870 stream      0      0 fffffe01c22c71d8        0        0        0 /var/run/cups.sock
fffffe010349ed20 stream      0      0 fffffe0103e66588        0        0        0 /tmp/.s.PGSQL.5432
fffffe01034a60f0 stream      0      0 fffffe0103457938        0        0        0 /var/run/rpcbind.sock
fffffe01034a63c0 stream      0      0 fffffe01034981d8        0        0        0 /var/run/devd.pipe
fffffe0103a5b780 dgram       0      0        0 fffffe0103517b40        0 fffffe010349b0f0
fffffe01035174b0 dgram       0      0        0 fffffe0103517c30        0 fffffe010349dc30
fffffe010349dc30 dgram       0      0        0 fffffe0103517c30        0 fffffe0103a5b960
fffffe0103a5b960 dgram       0      0        0 fffffe0103517c30        0        0
fffffe010349b0f0 dgram       0      0        0 fffffe0103517b40        0        0
fffffe0103517b40 dgram       0      0 fffffe01037f1000        0 fffffe0103a5b780        0 /var/run/logpriv
fffffe0103517c30 dgram       0      0 fffffe00c0b14ce8        0 fffffe01035174b0        0 /var/run/log

------------------------------------------------------------------------
netstat -aL

Current listen queue sizes (qlen/incqlen/maxqlen)
Proto Listen         Local Address 
tcp4  0/0/128        localhost.31416 
tcp4  0/0/50         *.bacula-dir 
tcp4  0/0/128        *.ssh 
tcp6  0/0/128        *.ssh 
tcp4  0/0/50         *.bacula-fd 
tcp4  0/0/50         *.bacula-sd 
tcp4  0/0/128        *.ipp 
tcp6  0/0/128        *.ipp 
tcp4  0/0/20         localhost.submission 
tcp4  0/0/20         localhost.smtp 
tcp4  0/0/20         borg.submission 
tcp4  0/0/20         borg.smtp 
tcp4  0/0/128        localhost.postgresql 
tcp6  0/0/128        localhost.postgresql 
tcp6  0/0/5          *.nfsd 
tcp4  0/0/5          *.nfsd 
tcp4  0/0/128        *.843 
tcp6  0/0/128        *.843 
tcp4  0/0/128        *.sunrpc 
tcp6  0/0/128        *.sunrpc 
unix  0/0/128        /var/run/cups.sock
unix  0/0/128        /tmp/.s.PGSQL.5432
unix  0/0/128        /var/run/rpcbind.sock
unix  0/0/4          /var/run/devd.pipe

------------------------------------------------------------------------
fstat

fstat: can't read file 1 at 0xfffffffffffffff
fstat: can't read file 2 at 0x78
fstat: can't read file 4 at 0xffff
fstat: can't read file 7 at 0xfffffffffffffff
fstat: can't read file 8 at 0x78
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read file 1 at 0xfffffffffffffff
fstat: can't read file 2 at 0x78
fstat: can't read file 4 at 0xffff
fstat: can't read file 7 at 0xfffffffffffffff
fstat: can't read file 8 at 0x78
fstat: can't read file 10 at 0xffff
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read file 1 at 0xfffffffffffffff
fstat: can't read file 2 at 0x78
fstat: can't read file 4 at 0xffff
fstat: can't read file 7 at 0xfffffffffffffff
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read file 1 at 0xfffffffffffffff
fstat: can't read file 2 at 0x78
fstat: can't read file 4 at 0xffff
fstat: can't read file 7 at 0xfffffffffffffff
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read file 1 at 0xfffffffffffffff
fstat: can't read file 2 at 0x78
fstat: can't read file 4 at 0xffff
fstat: can't read file 7 at 0xfffffffffffffff
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read file 1 at 0xfffffffffffffff
fstat: can't read file 2 at 0x78
fstat: can't read file 4 at 0xffff
fstat: can't read file 7 at 0xfffffffffffffff
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read file 1 at 0xfffffffffffffff
fstat: can't read file 2 at 0x78
fstat: can't read file 4 at 0xffff
fstat: can't read file 7 at 0xfffffffffffffff
fstat: can't read file 8 at 0x78
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read file 1 at 0xfffffffffffffff
fstat: can't read file 2 at 0x78
fstat: can't read file 4 at 0xffff
fstat: can't read file 7 at 0xfffffffffffffff
fstat: can't read file 8 at 0x78
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read file 1 at 0xfffffffffffffff
fstat: can't read file 2 at 0x78
fstat: can't read file 4 at 0xffff
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read file 1 at 0xfffffffffffffff
fstat: can't read file 2 at 0x78
fstat: can't read file 4 at 0xffff
fstat: can't read file 7 at 0xfffffffffffffff
fstat: can't read file 8 at 0x78
fstat: can't read file 10 at 0xffff
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read file 1 at 0xfffffffffffffff
fstat: can't read file 2 at 0x78
fstat: can't read file 4 at 0xffff
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read file 1 at 0xfffffffffffffff
fstat: can't read file 2 at 0x78
fstat: can't read file 4 at 0xffff
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read file 1 at 0xfffffffffffffff
fstat: can't read file 2 at 0x78
fstat: can't read file 4 at 0xffff
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read file 1 at 0xfffffffffffffff
fstat: can't read file 2 at 0x78
fstat: can't read file 4 at 0xffff
fstat: can't read file 7 at 0xfffffffffffffff
fstat: can't read file 8 at 0x78
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read file 1 at 0xfffffffffffffff
fstat: can't read file 2 at 0x78
fstat: can't read file 4 at 0xffff
fstat: can't read file 7 at 0xfffffffffffffff
fstat: can't read file 8 at 0x78
fstat: can't read file 10 at 0xffff
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read file 1 at 0xfffffffffffffff
fstat: can't read file 2 at 0x78
fstat: can't read file 4 at 0xffff
fstat: can't read file 7 at 0xfffffffffffffff
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read file 1 at 0xfffffffffffffff
fstat: can't read file 2 at 0x78
fstat: can't read file 4 at 0xffff
fstat: can't read file 7 at 0xfffffffffffffff
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read file 1 at 0xfffffffffffffff
fstat: can't read file 2 at 0x78
fstat: can't read file 4 at 0xffff
fstat: can't read file 7 at 0xfffffffffffffff
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read file 1 at 0xfffffffffffffff
fstat: can't read file 2 at 0x78
fstat: can't read file 4 at 0xffff
fstat: can't read file 7 at 0xfffffffffffffff
fstat: can't read file 8 at 0x78
fstat: can't read file 10 at 0xffff
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read file 1 at 0xfffffffffffffff
fstat: can't read file 2 at 0x78
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read file 1 at 0xfffffffffffffff
fstat: can't read file 2 at 0x78
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read file 1 at 0xfffffffffffffff
fstat: can't read file 2 at 0x78
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read file 1 at 0xfffffffffffffff
fstat: can't read file 2 at 0x78
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read file 1 at 0xfffffffffffffff
fstat: can't read file 2 at 0x78
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read file 1 at 0xfffffffffffffff
fstat: can't read file 2 at 0x78
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read file 1 at 0xfffffffffffffff
fstat: can't read file 2 at 0x78
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read file 1 at 0xfffffffffffffff
fstat: can't read file 2 at 0x78
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read file 1 at 0xfffffffffffffff
fstat: can't read file 2 at 0x78
fstat: can't read file 4 at 0xffff
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read file 1 at 0xfffffffffffffff
fstat: can't read file 2 at 0x78
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read file 1 at 0xfffffffffffffff
fstat: can't read file 2 at 0x78
fstat: can't read file 4 at 0xffff
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read file 1 at 0xfffffffffffffff
fstat: can't read file 2 at 0x78
fstat: can't read file 4 at 0xffff
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read file 1 at 0xfffffffffffffff
fstat: can't read file 2 at 0x78
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read file 1 at 0xfffffffffffffff
fstat: can't read file 2 at 0x78
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read file 1 at 0xfffffffffffffff
fstat: can't read file 2 at 0x78
fstat: can't read file 4 at 0xffff
fstat: can't read file 7 at 0xfffffffffffffff
fstat: can't read file 8 at 0x78
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read file 1 at 0xfffffffffffffff
fstat: can't read file 2 at 0x78
fstat: can't read file 4 at 0xffff
fstat: can't read file 7 at 0xfffffffffffffff
fstat: can't read file 8 at 0x78
fstat: can't read file 10 at 0xffff
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read file 1 at 0xfffffffffffffff
fstat: can't read file 2 at 0x78
fstat: can't read file 4 at 0xffff
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read file 1 at 0xfffffffffffffff
fstat: can't read file 2 at 0x78
fstat: can't read file 4 at 0xffff
fstat: can't read file 7 at 0xfffffffffffffff
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read file 1 at 0xfffffffffffffff
fstat: can't read file 2 at 0x78
fstat: can't read file 4 at 0xffff
fstat: can't read file 7 at 0xfffffffffffffff
fstat: can't read file 8 at 0x78
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read file 1 at 0xfffffffffffffff
fstat: can't read file 2 at 0x78
fstat: can't read file 4 at 0xffff
fstat: can't read file 7 at 0xfffffffffffffff
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read file 1 at 0xfffffffffffffff
fstat: can't read file 2 at 0x78
fstat: can't read file 4 at 0xffff
fstat: can't read file 7 at 0xfffffffffffffff
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read file 1 at 0xfffffffffffffff
fstat: can't read file 2 at 0x78
fstat: can't read file 4 at 0xffff
fstat: can't read file 7 at 0xfffffffffffffff
fstat: can't read file 8 at 0x78
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read file 1 at 0xfffffffffffffff
fstat: can't read file 2 at 0x78
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read file 1 at 0xfffffffffffffff
fstat: can't read file 2 at 0x78
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read file 1 at 0xfffffffffffffff
fstat: can't read file 2 at 0x78
fstat: can't read file 4 at 0xffff
fstat: can't read file 7 at 0xfffffffffffffff
fstat: can't read file 8 at 0x78
fstat: can't read file 10 at 0xffff
fstat: can't read file 13 at 0xfffffffffffffff
fstat: can't read file 14 at 0x78
fstat: can't read file 16 at 0xffff
fstat: can't read file 19 at 0xfffffffffffffff
fstat: can't read file 20 at 0x78
fstat: can't read file 22 at 0xffff
fstat: can't read file 23 at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read file 1 at 0xfffffffffffffff
fstat: can't read file 2 at 0x78
fstat: can't read file 4 at 0xffff
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read file 1 at 0xfffffffffffffff
fstat: can't read file 2 at 0x78
fstat: can't read file 4 at 0xffff
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read file 1 at 0xfffffffffffffff
fstat: can't read file 2 at 0x78
fstat: can't read file 4 at 0xffff
fstat: can't read file 7 at 0xfffffffffffffff
fstat: can't read file 8 at 0x78
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read file 1 at 0xfffffffffffffff
fstat: can't read file 2 at 0x78
fstat: can't read file 4 at 0xffff
fstat: can't read file 7 at 0xfffffffffffffff
fstat: can't read file 8 at 0x78
fstat: can't read file 10 at 0xffff
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read file 1 at 0xfffffffffffffff
fstat: can't read file 2 at 0x78
fstat: can't read file 4 at 0xffff
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read file 1 at 0xfffffffffffffff
fstat: can't read file 2 at 0x78
fstat: can't read file 4 at 0xffff
fstat: can't read file 7 at 0xfffffffffffffff
fstat: can't read file 8 at 0x78
fstat: can't read file 10 at 0xffff
fstat: can't read file 13 at 0xfffffffffffffff
fstat: can't read file 14 at 0x78
fstat: can't read file 16 at 0xffff
fstat: can't read file 19 at 0xfffffffffffffff
fstat: can't read file 20 at 0x78
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read file 1 at 0xfffffffffffffff
fstat: can't read file 2 at 0x78
fstat: can't read file 4 at 0xffff
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
USER     CMD          PID   FD MOUNT      INUM MODE         SZ|DV R/W
root     zfs         1044 root -         -       error    -
root     zfs         1044   wd -         -       error    -
root     zfs         1044 text -         -       error    -
root     zfs         1044    0* pipe fffffe00c0d6f000 <-> fffffe00c0d6f160  65224 rw
root     zfs         1044    6* pipe fffffe00c0d97730 <-> fffffe00c0d975d0      0 rw
root     csh         1042 root -         -       error    -
root     csh         1042   wd -         -       error    -
root     csh         1042 text -         -       error    -
root     sshd        1040 root -         -       error    -
root     sshd        1040   wd -         -       error    -
root     sshd        1040 text -         -       error    -
root     sshd        1040    0 /dev         28 crw-rw-rw-    null rw
root     sshd        1040    6 /dev         28 crw-rw-rw-    null rw
boinc    wcg_hpf2_rosetta_6  1038 root -         -       error    -
boinc    wcg_hpf2_rosetta_6  1038   wd -         -       error    -
boinc    wcg_hpf2_rosetta_6  1038 text -         -       error    -
boinc    wcg_hpf2_rosetta_6  1038    0 /dev         28 crw-rw-rw-    null rw
boinc    wcg_hpf2_rosetta_6  1038    6 -         -       error    -
boinc    wcg_hpf2_rosetta_6  1037 root -         -       error    -
boinc    wcg_hpf2_rosetta_6  1037   wd -         -       error    -
boinc    wcg_hpf2_rosetta_6  1037 text -         -       error    -
boinc    wcg_hpf2_rosetta_6  1037    0 /dev         28 crw-rw-rw-    null rw
boinc    wcg_hpf2_rosetta_6  1037    6 -         -       error    -
boinc    wcg_hpf2_rosetta_6  1036 root -         -       error    -
boinc    wcg_hpf2_rosetta_6  1036   wd -         -       error    -
boinc    wcg_hpf2_rosetta_6  1036 text -         -       error    -
boinc    wcg_hpf2_rosetta_6  1036    0 /dev         28 crw-rw-rw-    null rw
boinc    wcg_hpf2_rosetta_6  1036    6 -         -       error    -
boinc    wcg_hpf2_rosetta_6  1035 root -         -       error    -
boinc    wcg_hpf2_rosetta_6  1035   wd -         -       error    -
boinc    wcg_hpf2_rosetta_6  1035 text -         -       error    -
boinc    wcg_hpf2_rosetta_6  1035    0 /dev         28 crw-rw-rw-    null rw
boinc    wcg_hpf2_rosetta_6  1035    6 -         -       error    -
boinc    wcgrid_cep2_6.40_i  1034 root -         -       error    -
boinc    wcgrid_cep2_6.40_i  1034   wd -         -       error    -
boinc    wcgrid_cep2_6.40_i  1034 text -         -       error    -
boinc    wcgrid_cep2_6.40_i  1034    0 /dev         28 crw-rw-rw-    null rw
boinc    wcgrid_cep2_6.40_i  1034    6 -         -       error    -
boinc    wcgrid_cep2_6.40_i  1033 root -         -       error    -
boinc    wcgrid_cep2_6.40_i  1033   wd -         -       error    -
boinc    wcgrid_cep2_6.40_i  1033 text -         -       error    -
boinc    wcgrid_cep2_6.40_i  1033    0 /dev         28 crw-rw-rw-    null rw
boinc    wcgrid_cep2_6.40_i  1033    6 -         -       error    -
boinc    setiathome-6.12.am  1032 root -         -       error    -
boinc    setiathome-6.12.am  1032   wd -         -       error    -
boinc    setiathome-6.12.am  1032 text -         -       error    -
boinc    setiathome-6.12.am  1032    0 /dev         28 crw-rw-rw-    null rw
boinc    setiathome-6.12.am  1032    6 /dev         28 crw-rw-rw-    null rw
boinc    wcgrid_faah_7.15_i  1031 root -         -       error    -
boinc    wcgrid_faah_7.15_i  1031   wd -         -       error    -
boinc    wcgrid_faah_7.15_i  1031 text -         -       error    -
boinc    wcgrid_faah_7.15_i  1031    0 /dev         28 crw-rw-rw-    null rw
boinc    wcgrid_faah_7.15_i  1031    6 /dev         28 crw-rw-rw-    null rw
boinc    setiathome-6.12.am  1030 root -         -       error    -
boinc    setiathome-6.12.am  1030   wd -         -       error    -
boinc    setiathome-6.12.am  1030 text -         -       error    -
boinc    setiathome-6.12.am  1030    0 /dev         28 crw-rw-rw-    null rw
boinc    setiathome-6.12.am  1030    6 /dev         28 crw-rw-rw-    null rw
boinc    setiathome-6.12.am  1029 root -         -       error    -
boinc    setiathome-6.12.am  1029   wd -         -       error    -
boinc    setiathome-6.12.am  1029 text -         -       error    -
boinc    setiathome-6.12.am  1029    0 /dev         28 crw-rw-rw-    null rw
boinc    setiathome-6.12.am  1029    6 /dev         28 crw-rw-rw-    null rw
boinc    setiathome-6.12.am  1028 root -         -       error    -
boinc    setiathome-6.12.am  1028   wd -         -       error    -
boinc    setiathome-6.12.am  1028 text -         -       error    -
boinc    setiathome-6.12.am  1028    0 /dev         28 crw-rw-rw-    null rw
boinc    setiathome-6.12.am  1028    6 /dev         28 crw-rw-rw-    null rw
boinc    wcgrid_cep2_6.40_i  1027 root -         -       error    -
boinc    wcgrid_cep2_6.40_i  1027   wd -         -       error    -
boinc    wcgrid_cep2_6.40_i  1027 text -         -       error    -
boinc    wcgrid_cep2_6.40_i  1027    0 /dev         28 crw-rw-rw-    null rw
boinc    wcgrid_cep2_6.40_i  1027    6 -         -       error    -
boinc    wcgrid_faah_7.15_i  1026 root -         -       error    -
boinc    wcgrid_faah_7.15_i  1026   wd -         -       error    -
boinc    wcgrid_faah_7.15_i  1026 text -         -       error    -
boinc    wcgrid_faah_7.15_i  1026    0 /dev         28 crw-rw-rw-    null rw
boinc    wcgrid_faah_7.15_i  1026    6 /dev         28 crw-rw-rw-    null rw
boinc    wcg_hpf2_rosetta_6  1025 root -         -       error    -
boinc    wcg_hpf2_rosetta_6  1025   wd -         -       error    -
boinc    wcg_hpf2_rosetta_6  1025 text -         -       error    -
boinc    wcg_hpf2_rosetta_6  1025    0 /dev         28 crw-rw-rw-    null rw
boinc    wcg_hpf2_rosetta_6  1025    6 -         -       error    -
boinc    wcg_hpf2_rosetta_6  1024 root -         -       error    -
boinc    wcg_hpf2_rosetta_6  1024   wd -         -       error    -
boinc    wcg_hpf2_rosetta_6  1024 text -         -       error    -
boinc    wcg_hpf2_rosetta_6  1024    0 /dev         28 crw-rw-rw-    null rw
boinc    wcg_hpf2_rosetta_6  1024    6 -         -       error    -
root     ssh         1022 root -         -       error    -
root     ssh         1022   wd -         -       error    -
root     ssh         1022 text -         -       error    -
root     ssh         1022 ctty /dev         71 crw-------   ttyv0 rw
root     ssh         1022    0 /dev         71 crw-------   ttyv0 rw
root     ssh         1022    6 /dev         71 crw-------   ttyv0 rw
root     sh          1020 root -         -       error    -
root     sh          1020   wd -         -       error    -
root     sh          1020 text -         -       error    -
root     sh          1020 ctty /dev         71 crw-------   ttyv0 rw
root     sh          1020    0 /dev         71 crw-------   ttyv0 rw
root     sh          1020    6 /dev         71 crw-------   ttyv0 rw
root     getty       1017 root -         -       error    -
root     getty       1017   wd -         -       error    -
root     getty       1017 text -         -       error    -
root     getty       1017 ctty /dev         78 crw-------   ttyv7 rw
root     getty       1017    0 /dev         78 crw-------   ttyv7 rw
root     getty       1016 root -         -       error    -
root     getty       1016   wd -         -       error    -
root     getty       1016 text -         -       error    -
root     getty       1016 ctty /dev         77 crw-------   ttyv6 rw
root     getty       1016    0 /dev         77 crw-------   ttyv6 rw
root     getty       1015 root -         -       error    -
root     getty       1015   wd -         -       error    -
root     getty       1015 text -         -       error    -
root     getty       1015 ctty /dev         76 crw-------   ttyv5 rw
root     getty       1015    0 /dev         76 crw-------   ttyv5 rw
root     getty       1014 root -         -       error    -
root     getty       1014   wd -         -       error    -
root     getty       1014 text -         -       error    -
root     getty       1014 ctty /dev         75 crw-------   ttyv4 rw
root     getty       1014    0 /dev         75 crw-------   ttyv4 rw
root     getty       1013 root -         -       error    -
root     getty       1013   wd -         -       error    -
root     getty       1013 text -         -       error    -
root     getty       1013 ctty /dev         74 crw-------   ttyv3 rw
root     getty       1013    0 /dev         74 crw-------   ttyv3 rw
root     getty       1012 root -         -       error    -
root     getty       1012   wd -         -       error    -
root     getty       1012 text -         -       error    -
root     getty       1012 ctty /dev         73 crw-------   ttyv2 rw
root     getty       1012    0 /dev         73 crw-------   ttyv2 rw
root     getty       1011 root -         -       error    -
root     getty       1011   wd -         -       error    -
root     getty       1011 text -         -       error    -
root     getty       1011 ctty /dev         72 crw-------   ttyv1 rw
root     getty       1011    0 /dev         72 crw-------   ttyv1 rw
root     login       1010 root -         -       error    -
root     login       1010   wd -         -       error    -
root     login       1010 text -         -       error    -
root     login       1010 ctty /dev         71 crw-------   ttyv0 rw
root     login       1010    0 /dev         71 crw-------   ttyv0 rw
root     inetd        990 root -         -       error    -
root     inetd        990   wd -         -       error    -
root     inetd        990 text -         -       error    -
root     inetd        990    0 /dev         28 crw-rw-rw-    null rw
root     inetd        990    6 /dev         28 crw-rw-rw-    null rw
root     cron         969 root -         -       error    -
root     cron         969   wd -         -       error    -
root     cron         969 text -         -       error    -
root     cron         969    0 /dev         28 crw-rw-rw-    null rw
root     sshd         959 root -         -       error    -
root     sshd         959   wd -         -       error    -
root     sshd         959 text -         -       error    -
root     sshd         959    0 /dev         28 crw-rw-rw-    null rw
bacula   bacula-dir   954 root -         -       error    -
bacula   bacula-dir   954   wd -         -       error    -
bacula   bacula-dir   954 text -         -       error    -
bacula   bacula-dir   954    0 /dev         28 crw-rw-rw-    null  r
root     bacula-fd    951 root -         -       error    -
root     bacula-fd    951   wd -         -       error    -
root     bacula-fd    951 text -         -       error    -
root     bacula-fd    951    0 /dev         28 crw-rw-rw-    null  r
bacula   bacula-sd    948 root -         -       error    -
bacula   bacula-sd    948   wd -         -       error    -
bacula   bacula-sd    948 text -         -       error    -
bacula   bacula-sd    948    0 /dev         28 crw-rw-rw-    null  r
boinc    boinc_client   944 root -         -       error    -
boinc    boinc_client   944   wd -         -       error    -
boinc    boinc_client   944 text -         -       error    -
boinc    boinc_client   944    0 /dev         28 crw-rw-rw-    null rw
boinc    boinc_client   944    6 -         -       error    -
root     cupsd        938 root -         -       error    -
root     cupsd        938   wd -         -       error    -
root     cupsd        938 text -         -       error    -
root     cupsd        938    0 /dev         28 crw-rw-rw-    null  r
root     cupsd        938    6 /dev         28 crw-rw-rw-    null  w
root     cupsd        938   12 /dev         28 crw-rw-rw-    null  w
mailnull exim-4.80.1-2   928 root -         -       error    -
mailnull exim-4.80.1-2   928   wd -         -       error    -
mailnull exim-4.80.1-2   928 text -         -       error    -
mailnull exim-4.80.1-2   928    0 /dev         28 crw-rw-rw-    null rw
mailnull exim-4.80.1-2   928    6 /dev         28 crw-rw-rw-    null rw
pgsql    postgres     906 root -         -       error    -
pgsql    postgres     906   wd -         -       error    -
pgsql    postgres     906 text -         -       error    -
pgsql    postgres     906    0 /dev         28 crw-rw-rw-    null  r
pgsql    postgres     906    6 -         -       error    -
pgsql    postgres     905 root -         -       error    -
pgsql    postgres     905   wd -         -       error    -
pgsql    postgres     905 text -         -       error    -
pgsql    postgres     905    0 /dev         28 crw-rw-rw-    null  r
pgsql    postgres     905    6 -         -       error    -
pgsql    postgres     904 root -         -       error    -
pgsql    postgres     904   wd -         -       error    -
pgsql    postgres     904 text -         -       error    -
pgsql    postgres     904    0 /dev         28 crw-rw-rw-    null  r
pgsql    postgres     904    6 -         -       error    -
pgsql    postgres     903 root -         -       error    -
pgsql    postgres     903   wd -         -       error    -
pgsql    postgres     903 text -         -       error    -
pgsql    postgres     903    0 /dev         28 crw-rw-rw-    null  r
pgsql    postgres     903    6 -         -       error    -
pgsql    postgres     899 root -         -       error    -
pgsql    postgres     899   wd -         -       error    -
pgsql    postgres     899 text -         -       error    -
pgsql    postgres     899    0 /dev         28 crw-rw-rw-    null  r
pgsql    postgres     899    6 -         -       error    -
root     smartd       890 root -         -       error    -
root     smartd       890   wd -         -       error    -
root     smartd       890 text -         -       error    -
root     smartd       890    0 /dev         28 crw-rw-rw-    null rw
root     perl         886 root -         -       error    -
root     perl         886   wd -         -       error    -
root     perl         886 text -         -       error    -
root     perl         886    0 -         -       error    -
root     ng_queue     874 root -         -       error    -
root     ng_queue     874   wd -         -       error    -
root     ntpd         853 root -         -       error    -
root     ntpd         853   wd -         -       error    -
root     ntpd         853 text -         -       error    -
root     ntpd         853    0 /dev         28 crw-rw-rw-    null rw
root     ntpd         853    6 /dev         28 crw-rw-rw-    null rw
root     ntpd         853   12 /dev         28 crw-rw-rw-    null rw
root     ntpd         853   18* local dgram fffffe010349b0f0 <-> fffffe0103517b40
root     nfsd         817 root -         -       error    -
root     nfsd         817   wd -         -       error    -
root     nfsd         817 text -         -       error    -
root     nfsd         817    0 /dev         28 crw-rw-rw-    null rw
root     nfsd         815 root -         -       error    -
root     nfsd         815   wd -         -       error    -
root     nfsd         815 text -         -       error    -
root     nfsd         815    0 /dev         28 crw-rw-rw-    null rw
root     nfsd         815    6 /dev         28 crw-rw-rw-    null rw
root     mountd       809 root -         -       error    -
root     mountd       809   wd -         -       error    -
root     mountd       809 text -         -       error    -
root     mountd       809    0 /dev         28 crw-rw-rw-    null rw
root     mountd       809    6 /dev         28 crw-rw-rw-    null rw
root     rpcbind      774 root -         -       error    -
root     rpcbind      774   wd -         -       error    -
root     rpcbind      774 text -         -       error    -
root     rpcbind      774    0 /dev         28 crw-rw-rw-    null rw
root     rpcbind      774    6 /dev         28 crw-rw-rw-    null rw
root     watchdogd    760 root -         -       error    -
root     watchdogd    760   wd -         -       error    -
root     watchdogd    760 text -         -       error    -
root     watchdogd    760    0 /dev         28 crw-rw-rw-    null rw
root     syslogd      757 root -         -       error    -
root     syslogd      757   wd -         -       error    -
root     syslogd      757 text -         -       error    -
root     syslogd      757    0 /dev         28 crw-rw-rw-    null rw
root     syslogd      757    6 /dev         28 crw-rw-rw-    null rw
root     syslogd      757   12 /dev         28 crw-rw-rw-    null rw
root     syslogd      757   18 -         -       error    -
root     devd         632 root -         -       error    -
root     devd         632   wd -         -       error    -
root     devd         632 text -         -       error    -
root     devd         632    0 /dev         28 crw-rw-rw-    null rw
root     init           1 root -         -       error    -
root     init           1   wd -         -       error    -
root     init           1 text -         -       error    -
root     kernel         0 root -         -       error    -
root     kernel         0   wd -         -       error    -

------------------------------------------------------------------------
dmesg

Copyright (c) 1992-2013 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
 	The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 10.0-CURRENT #129 r248695: Mon Mar 25 05:03:32 CDT 2013
     root_at_borg.lerctr.org:/usr/obj/usr/src/sys/BORG-DTRACE amd64
FreeBSD clang version 3.2 (tags/RELEASE_32/final 170710) 20121221
CPU: Intel(R) Xeon(R) CPU           E5410  _at_ 2.33GHz (2327.54-MHz K8-class CPU)
   Origin = "GenuineIntel"  Id = 0x10676  Family = 0x6  Model = 0x17  Stepping = 6
   Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
   Features2=0xce3bd<SSE3,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,SSE4.1>
   AMD Features=0x20100800<SYSCALL,NX,LM>
   AMD Features2=0x1<LAHF>
   TSC: P-state invariant, performance statistics
real memory  = 68719476736 (65536 MB)
avail memory = 63193878528 (60266 MB)
Event timer "LAPIC" quality 400
ACPI APIC Table: <PTLTD  	 APIC  >
FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs
FreeBSD/SMP: 2 package(s) x 4 core(s)
  cpu0 (BSP): APIC ID:  0
  cpu1 (AP): APIC ID:  1
  cpu2 (AP): APIC ID:  2
  cpu3 (AP): APIC ID:  3
  cpu4 (AP): APIC ID:  4
  cpu5 (AP): APIC ID:  5
  cpu6 (AP): APIC ID:  6
  cpu7 (AP): APIC ID:  7
ioapic0 <Version 2.0> irqs 0-23 on motherboard
ioapic1 <Version 2.0> irqs 24-47 on motherboard
kbd1 at kbdmux0
netmap: loaded module
cryptosoft0: <software crypto> on motherboard
acpi0: <PTLTD   RSDT> on motherboard
acpi0: Power Button (fixed)
unknown: I/O range not supported
cpu0: <ACPI CPU> on acpi0
cpu1: <ACPI CPU> on acpi0
cpu2: <ACPI CPU> on acpi0
cpu3: <ACPI CPU> on acpi0
cpu4: <ACPI CPU> on acpi0
cpu5: <ACPI CPU> on acpi0
cpu6: <ACPI CPU> on acpi0
cpu7: <ACPI CPU> on acpi0
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff irq 0,8 on acpi0
Timecounter "HPET" frequency 14318180 Hz quality 950
Event timer "HPET" frequency 14318180 Hz quality 350
Event timer "HPET1" frequency 14318180 Hz quality 340
Event timer "HPET2" frequency 14318180 Hz quality 340
atrtc0: <AT realtime clock> port 0x70-0x71 on acpi0
Event timer "RTC" frequency 32768 Hz quality 0
attimer0: <AT timer> port 0x40-0x43,0x50-0x53 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> at device 2.0 on pci0
pci1: <ACPI PCI bus> on pcib1
pcib2: <ACPI PCI-PCI bridge> irq 16 at device 0.0 on pci1
pci2: <ACPI PCI bus> on pcib2
pcib3: <ACPI PCI-PCI bridge> irq 16 at device 0.0 on pci2
pci3: <ACPI PCI bus> on pcib3
pcib4: <ACPI PCI-PCI bridge> at device 0.0 on pci3
pci4: <ACPI PCI bus> on pcib4
pcib5: <ACPI PCI-PCI bridge> at device 0.2 on pci3
pci5: <ACPI PCI bus> on pcib5
pcib6: <ACPI PCI-PCI bridge> irq 18 at device 2.0 on pci2
pci6: <ACPI PCI bus> on pcib6
em0: <Intel(R) PRO/1000 Network Connection 7.3.7> port 0x2000-0x201f mem 0xd8020000-0xd803ffff,0xd8000000-0xd801ffff irq 18 at device 0.0 on pci6
em0: Using an MSI interrupt
em0: Ethernet address: 00:30:48:f2:29:9c
001.000008 netmap_attach [1680] success for em0
em1: <Intel(R) PRO/1000 Network Connection 7.3.7> port 0x2020-0x203f mem 0xd8060000-0xd807ffff,0xd8040000-0xd805ffff irq 19 at device 0.1 on pci6
em1: Using an MSI interrupt
em1: Ethernet address: 00:30:48:f2:29:9d
001.000009 netmap_attach [1680] success for em1
pcib7: <ACPI PCI-PCI bridge> at device 0.3 on pci1
pci7: <ACPI PCI bus> on pcib7
pcib8: <ACPI PCI-PCI bridge> at device 4.0 on pci0
pci8: <ACPI PCI bus> on pcib8
pcib9: <ACPI PCI-PCI bridge> at device 6.0 on pci0
pci9: <ACPI PCI bus> on pcib9
pci0: <base peripheral> at device 8.0 (no driver attached)
pcib10: <ACPI PCI-PCI bridge> irq 17 at device 28.0 on pci0
pci10: <ACPI PCI bus> on pcib10
uhci0: <Intel 631XESB/632XESB/3100 USB controller USB-1> port 0x1800-0x181f irq 17 at device 29.0 on pci0
usbus0 on uhci0
uhci1: <Intel 631XESB/632XESB/3100 USB controller USB-2> port 0x1820-0x183f irq 19 at device 29.1 on pci0
usbus1 on uhci1
uhci2: <Intel 631XESB/632XESB/3100 USB controller USB-3> port 0x1840-0x185f irq 18 at device 29.2 on pci0
usbus2 on uhci2
ehci0: <Intel 63XXESB USB 2.0 controller> mem 0xd8500400-0xd85007ff irq 17 at device 29.7 on pci0
usbus3: EHCI version 1.0
usbus3 on ehci0
pcib11: <ACPI PCI-PCI bridge> at device 30.0 on pci0
pci11: <ACPI PCI bus> on pcib11
vgapci0: <VGA-compatible display> port 0x3000-0x30ff mem 0xd0000000-0xd7ffffff,0xd8200000-0xd820ffff irq 18 at device 1.0 on pci11
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel 63XXESB2 UDMA100 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x1860-0x186f at device 31.1 on pci0
ata0: <ATA channel> at channel 0 on atapci0
ahci0: <Intel ESB2 AHCI SATA controller> port 0x18a0-0x18a7,0x1874-0x1877,0x1878-0x187f,0x1870-0x1873,0x1880-0x189f mem 0xd8500800-0xd8500bff irq 19 at device 31.2 on pci0
ahci0: AHCI v1.10 with 6 3Gbps ports, Port Multiplier supported
ahcich0: <AHCI channel> at channel 0 on ahci0
ahcich1: <AHCI channel> at channel 1 on ahci0
ahcich2: <AHCI channel> at channel 2 on ahci0
ahcich3: <AHCI channel> at channel 3 on ahci0
ahcich4: <AHCI channel> at channel 4 on ahci0
ahcich5: <AHCI channel> at channel 5 on ahci0
ichsmb0: <Intel 631xESB/6321ESB (ESB2) SMBus controller> port 0x1100-0x111f irq 19 at device 31.3 on pci0
smbus0: <System Management Bus> on ichsmb0
acpi_button0: <Power Button> on acpi0
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0
fdc0: <floppy drive controller> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0
fd0: <1440-KB 3.5" drive> on fdc0 drive 0
ppc0: <Parallel port> port 0x378-0x37f,0x778-0x77f irq 7 drq 3 on acpi0
ppc0: SMC-like chipset (ECP/EPP/PS2/NIBBLE) in COMPATIBLE mode
ppc0: FIFO with 16/16/9 bytes threshold
ppbus0: <Parallel port bus> on ppc0
lpt0: <Printer> on ppbus0
lpt0: Interrupt-driven port
ppi0: <Parallel I/O> on ppbus0
ichwd0 on isa0
orm0: <ISA Option ROM> at iomem 0xc0000-0xcafff on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
coretemp0: <CPU On-Die Thermal Sensors> on cpu0
est0: <Enhanced SpeedStep Frequency Control> on cpu0
p4tcc0: <CPU Frequency Thermal Control> on cpu0
coretemp1: <CPU On-Die Thermal Sensors> on cpu1
est1: <Enhanced SpeedStep Frequency Control> on cpu1
p4tcc1: <CPU Frequency Thermal Control> on cpu1
coretemp2: <CPU On-Die Thermal Sensors> on cpu2
est2: <Enhanced SpeedStep Frequency Control> on cpu2
p4tcc2: <CPU Frequency Thermal Control> on cpu2
coretemp3: <CPU On-Die Thermal Sensors> on cpu3
est3: <Enhanced SpeedStep Frequency Control> on cpu3
p4tcc3: <CPU Frequency Thermal Control> on cpu3
coretemp4: <CPU On-Die Thermal Sensors> on cpu4
est4: <Enhanced SpeedStep Frequency Control> on cpu4
p4tcc4: <CPU Frequency Thermal Control> on cpu4
coretemp5: <CPU On-Die Thermal Sensors> on cpu5
est5: <Enhanced SpeedStep Frequency Control> on cpu5
p4tcc5: <CPU Frequency Thermal Control> on cpu5
coretemp6: <CPU On-Die Thermal Sensors> on cpu6
est6: <Enhanced SpeedStep Frequency Control> on cpu6
p4tcc6: <CPU Frequency Thermal Control> on cpu6
coretemp7: <CPU On-Die Thermal Sensors> on cpu7
est7: <Enhanced SpeedStep Frequency Control> on cpu7
p4tcc7: <CPU Frequency Thermal Control> on cpu7
ZFS filesystem version: 5
ZFS storage pool version: features support (5000)
Timecounters tick every 1.000 msec
vboxdrv: fAsync=0 offMin=0x40c offMax=0x58d
usbus0: 12Mbps Full Speed USB v1.0
usbus1: 12Mbps Full Speed USB v1.0
usbus2: 12Mbps Full Speed USB v1.0
usbus3: 480Mbps High Speed USB v2.0
ugen1.1: <Intel> at usbus1
uhub0: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus1
ugen0.1: <Intel> at usbus0
uhub1: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0
ugen3.1: <Intel> at usbus3
uhub2: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus3
ugen2.1: <Intel> at usbus2
uhub3: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus2
ata0: DMA limited to UDMA33, controller found non-ATA66 cable
ada0 at ahcich0 bus 0 scbus1 target 0 lun 0
ada0: <ST2000DL003-9VT166 CC3D> ATA-8 SATA 3.x device
ada0: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada0: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
ada0: Previously was known as ad4
ada1 at ahcich1 bus 0 scbus2 target 0 lun 0
ada1: <ST2000DL003-9VT166 CC32> ATA-8 SATA 3.x device
ada1: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada1: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
ada1: Previously was known as ad6
ada2 at ahcich2 bus 0 scbus3 target 0 lun 0
ada2: <ST2000DL003-9VT166 CC32> ATA-8 SATA 3.x device
ada2: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada2: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
ada2: Previously was known as ad8
ada3 at ahcich3 bus 0 scbus4 target 0 lun 0
ada3: <ST2000DL003-9VT166 CC32> ATA-8 SATA 3.x device
ada3: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada3: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
ada3: Previously was known as ad10
ada4 at ahcich4 bus 0 scbus5 target 0 lun 0
ada4: <ST2000DL003-9VT166 CC32> ATA-8 SATA 3.x device
ada4: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada4: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
ada4: Previously was known as ad12
ada5 at ahcich5 bus 0 scbus6 target 0 lun 0
ada5: <ST2000DL003-9VT166 CC3D> ATA-8 SATA 3.x device
ada5: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada5: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
ada5: Previously was known as ad14
cd0 at ata0 bus 0 scbus0 target 0 lun 0
cd0: <Memorex DVD+-RAM 510L v1 MWS7> Removable CD-ROM SCSI-0 device 
cd0: 33.300MB/s transfers (UDMA2, ATAPI 12bytes, PIO 65534bytes)
cd0: Attempt to query device size failed: NOT READY, Medium not present
SMP: AP CPU #7 Launched!
SMP: AP CPU #1 Launched!
SMP: AP CPU #2 Launched!
SMP: AP CPU #3 Launched!
SMP: AP CPU #4 Launched!
SMP: AP CPU #6 Launched!
SMP: AP CPU #5 Launched!
uhub1: 2 ports with 2 removable, self powered
uhub0: 2 ports with 2 removable, self powered
uhub3: 2 ports with 2 removable, self powered
Root mount waiting for: usbus3
uhub2: 6 ports with 6 removable, self powered
Root mount waiting for: usbus3
ugen3.2: <Peppercon AG> at usbus3
ukbd0: <Peppercon AG Multidevice, class 0/0, rev 2.00/0.01, addr 2> on usbus3
kbd2 at ukbd0
Trying to mount root from zfs:zroot/ROOT/default []...
ffclock reset: HPET (14318180 Hz), time = 1364219053.500000000
Setting hostuuid: 53d19f64-d663-a017-8922-0030488e9ff3.
Setting hostid: 0xf53a926e.
Entropy harvesting: interrupts ethernet point_to_point kickstart.
Starting file system checks:
Mounting local file systems:.
Writing entropy file:.
Setting hostname: borg.lerctr.org.
Starting Network: lo0 em0 em1.
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
 	options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
 	inet6 ::1 prefixlen 128
 	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
 	inet 127.0.0.1 netmask 0xff000000
 	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
 	options=4219b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_MAGIC,VLAN_HWTSO>
 	ether 00:30:48:f2:29:9c
 	inet 192.168.200.4 netmask 0xffffff00 broadcast 192.168.200.255
 	inet6 fe80::230:48ff:fef2:299c%em0 prefixlen 64 scopeid 0x1
 	nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
 	media: Ethernet autoselect
 	status: no carrier
em1: flags=8c02<BROADCAST,OACTIVE,SIMPLEX,MULTICAST> metric 0 mtu 1500
 	options=4219b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_MAGIC,VLAN_HWTSO>
 	ether 00:30:48:f2:29:9d
 	nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
 	media: Ethernet autoselect
 	status: no carrier
Starting devd.
Starting Network: em1.
em1: flags=8c02<BROADCAST,OACTIVE,SIMPLEX,MULTICAST> metric 0 mtu 1500
 	options=4219b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_MAGIC,VLAN_HWTSO>
 	ether 00:30:48:f2:29:9d
 	nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
 	media: Ethernet autoselect
 	status: no carrier
uhid0: <Peppercon AG Multidevice, class 0/0, rev 2.00/0.01, addr 2> on usbus3
add net default: gateway 192.168.200.11
add net ::ffff:0.0.0.0: gateway ::1
add net ::0.0.0.0: gateway ::1
add net fe80::: gateway ::1
add net ff02::: gateway ::1
Mounting NFS file systems:.
ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib /usr/local/kde4/lib /usr/local/lib/compat /usr/local/lib/dbmail /usr/local/lib/event2 /usr/local/lib/pth /usr/local/lib/qt4 /usr/local/lib/virtualbox
32-bit compatibility ldconfig path: /usr/lib32 /usr/local/lib32/compat
Creating and/or trimming log files.
Starting syslogd.
Starting watchdogd.
No core dumps found.
Additional ABI support: linux.
Starting rpcbind.
NFS access cache time=60
Clearing /tmp (X related).
Starting mountd.
NFSv4 is disabled
Starting nfsd.
Updating motd:.
Starting ntpd.
Starting sshblock.
Starting smartd.
Updating cpucodes...
/usr/local/share/cpucontrol/m401067660F.fw: updating cpu /dev/cpuctl0 from rev 0x60c to rev 0x60f... done.
/usr/local/share/cpucontrol/m401067660F.fw: updating cpu /dev/cpuctl1 from rev 0x60c to rev 0x60f... done.
/usr/local/share/cpucontrol/m401067660F.fw: updating cpu /dev/cpuctl2 from rev 0x60c to rev 0x60f... done.
/usr/local/share/cpucontrol/m401067660F.fw: updating cpu /dev/cpuctl3 from rev 0x60c to rev 0x60f... done.
/usr/local/share/cpucontrol/m401067660F.fw: updating cpu /dev/cpuctl4 from rev 0x60c to rev 0x60f... done.
/usr/local/share/cpucontrol/m401067660F.fw: updating cpu /dev/cpuctl5 from rev 0x60c to rev 0x60f... done.
/usr/local/share/cpucontrol/m401067660F.fw: updating cpu /dev/cpuctl6 from rev 0x60c to rev 0x60f... done.
/usr/local/share/cpucontrol/m401067660F.fw: updating cpu /dev/cpuctl7 from rev 0x60c to rev 0x60f... done.
Done.
Starting exim.
Starting cupsd.
Starting boinc_client.
Starting bacula_sd.
Starting bacula_fd.
Starting bacula_dir.
Performing sanity check on sshd configuration.
Starting sshd.
Configuring syscons: blanktime.
Starting cron.
Starting inetd.
Starting background file system checks in 60 seconds.

Mon Mar 25 08:44:56 CDT 2013
Mar 25 08:45:03 borg login: ROOT LOGIN (root) ON ttyv0


Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 00
fault virtual address	= 0x378
fault code		= supervisor read data, page not present
instruction pointer	= 0x20:0xffffffff80531426
stack pointer	        = 0x28:0xffffff91579193d0
frame pointer	        = 0x28:0xffffff9157919470
code segment		= base 0x0, limit 0xfffff, type 0x1b
 			= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags	= interrupt enabled, resume, IOPL = 0
current process		= 1044 (zfs)
trap number		= 12
panic: page fault
cpuid = 0
Uptime: 2m10s
Dumping 4913 out of 64747 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91%

------------------------------------------------------------------------
kernel config

options	CONFIG_AUTOGENERATED
ident	BORG-DTRACE
machine	amd64
cpu	HAMMER
makeoptions	WITH_CTF=1
makeoptions	DEBUG=-g
options	FFCLOCK
options	USB_DEBUG
options	RDRAND_RNG
options	PADLOCK_RNG
options	ATH_ENABLE_11N
options	AH_AR5416_INTERRUPT_MITIGATION
options	AH_SUPPORT_AR5416
options	IEEE80211_SUPPORT_MESH
options	IEEE80211_AMPDU_AGE
options	IEEE80211_DEBUG
options	SC_PIXEL_MODE
options	VESA
options	CTL_DISABLE
options	AHD_REG_PRETTY_PRINT
options	AHC_REG_PRETTY_PRINT
options	ATA_STATIC_ID
options	ATA_CAM
options	SMP
options	MALLOC_DEBUG_MAXZONES=8
options	INCLUDE_CONFIG_FILE
options	DDB_CTF
options	KDTRACE_HOOKS
options	KDTRACE_FRAME
options	MAC
options	CAPABILITIES
options	CAPABILITY_MODE
options	AUDIT
options	HWPMC_HOOKS
options	KBD_INSTALL_CDEV
options	PRINTF_BUFR_SIZE=128
options	_KPOSIX_PRIORITY_SCHEDULING
options	SYSVSEM
options	SYSVMSG
options	SYSVSHM
options	STACK
options	KTRACE
options	SCSI_DELAY=5000
options	COMPAT_FREEBSD7
options	COMPAT_FREEBSD6
options	COMPAT_FREEBSD5
options	COMPAT_FREEBSD4
options	COMPAT_FREEBSD32
options	GEOM_LABEL
options	GEOM_RAID
options	GEOM_PART_GPT
options	PSEUDOFS
options	PROCFS
options	CD9660
options	MSDOSFS
options	NFS_ROOT
options	NFSLOCKD
options	NFSD
options	NFSCL
options	QUOTA
options	SCTP
options	TCP_OFFLOAD
options	INET6
options	INET
options	PREEMPTION
options	SCHED_ULE
options	NEW_PCIB
options	GEOM_PART_MBR
options	GEOM_PART_EBR_COMPAT
options	GEOM_PART_EBR
options	GEOM_PART_BSD
device	isa
device	mem
device	io
device	uart_ns8250
device	cpufreq
device	acpi
device	pci
device	fdc
device	ahci
device	ata
device	esp
device	isci
device	scbus
device	ch
device	da
device	sa
device	cd
device	pass
device	ses
device	ctl
device	atkbdc
device	atkbd
device	psm
device	kbdmux
device	vga
device	splash
device	sc
device	agp
device	uart
device	ppc
device	ppbus
device	lpt
device	ppi
device	em
device	miibus
device	cas
device	gem
device	hme
device	nfe
device	nge
device	loop
device	random
device	ether
device	vlan
device	tun
device	md
device	gif
device	faith
device	firmware
device	bpf
device	uhci
device	ohci
device	ehci
device	xhci
device	usb
device	ukbd
device	umass
device	virtio
device	virtio_pci
device	vtnet
device	virtio_blk
device	virtio_scsi
device	virtio_balloon
device	netmap

------------------------------------------------------------------------
ddb capture buffer

ddb: ddb_capture: kvm_nlist

-- 
Larry Rosenman                     http://www.lerctr.org/~ler
Phone: +1 512-248-2683                 E-Mail: ler_at_lerctr.org
US Mail: 430 Valona Loop, Round Rock, TX 78681-3893
Received on Mon Mar 25 2013 - 13:01:45 UTC

This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:40:36 UTC