ZFS related Crash.

From: Larry Rosenman <ler_at_lerctr.org>
Date: Sat, 10 Aug 2013 21:18:44 -0500 (CDT)
r254196 -- this was on the reboot to reboot to the new kernel.

I've seen a similar crash today.

What else do folks need?

I can provide the vmcore and ssh access.


borg.lerctr.org dumped core - see /var/crash/vmcore.0

Sat Aug 10 21:14:09 CDT 2013

FreeBSD borg.lerctr.org 10.0-CURRENT FreeBSD 10.0-CURRENT #17: Sat Aug 10 14:10:47 CDT 2013     root_at_borg.lerctr.org:/usr/obj/usr/src/sys/BORG-DTRACE  amd64

panic: general protection fault

GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd"...

Unread portion of the kernel message buffer:
<118>.
<118>Terminated
<118>Aug 10 21:05:44 borg syslogd: exiting on signal 15
Waiting (max 60 seconds) for system process `vnlru' to stop...done
Waiting (max 60 seconds) for system process `bufdaemon' to stop...done

Waiting (max 60 seconds) for system process `syncer' to stop...Syncing disks, vnodes remaining...0 0 0 0 0 0 0 done
All buffers synced.


Fatal trap 9: general protection fault while in kernel mode
cpuid = 0; apic id = 00
instruction pointer	= 0x20:0xffffffff80568be0
stack pointer	        = 0x28:0xffffff8fcb587600
frame pointer	        = 0x28:0xffffff8fcb587610
code segment		= base 0x0, limit 0xfffff, type 0x1b
 			= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags	= interrupt enabled, resume, IOPL = 0
current process		= 1 (init)
trap number		= 9
panic: general protection fault
cpuid = 0
Uptime: 1h1m10s
Dumping 7633 out of 64479 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91%

Reading symbols from /boot/kernel.old/zfs.ko.symbols...done.
Loaded symbols for /boot/kernel.old/zfs.ko.symbols
Reading symbols from /boot/kernel.old/acl_nfs4.ko.symbols...done.
Loaded symbols for /boot/kernel.old/acl_nfs4.ko.symbols
Reading symbols from /boot/kernel.old/opensolaris.ko.symbols...done.
Loaded symbols for /boot/kernel.old/opensolaris.ko.symbols
Reading symbols from /boot/kernel.old/linux.ko.symbols...done.
Loaded symbols for /boot/kernel.old/linux.ko.symbols
Reading symbols from /boot/kernel.old/snd_envy24ht.ko.symbols...done.
Loaded symbols for /boot/kernel.old/snd_envy24ht.ko.symbols
Reading symbols from /boot/kernel.old/snd_spicds.ko.symbols...done.
Loaded symbols for /boot/kernel.old/snd_spicds.ko.symbols
Reading symbols from /boot/kernel.old/sound.ko.symbols...done.
Loaded symbols for /boot/kernel.old/sound.ko.symbols
Reading symbols from /boot/kernel.old/coretemp.ko.symbols...done.
Loaded symbols for /boot/kernel.old/coretemp.ko.symbols
Reading symbols from /boot/kernel.old/ichsmb.ko.symbols...done.
Loaded symbols for /boot/kernel.old/ichsmb.ko.symbols
Reading symbols from /boot/kernel.old/smbus.ko.symbols...done.
Loaded symbols for /boot/kernel.old/smbus.ko.symbols
Reading symbols from /boot/kernel.old/ichwd.ko.symbols...done.
Loaded symbols for /boot/kernel.old/ichwd.ko.symbols
Reading symbols from /boot/kernel.old/cpuctl.ko.symbols...done.
Loaded symbols for /boot/kernel.old/cpuctl.ko.symbols
Reading symbols from /boot/kernel.old/crypto.ko.symbols...done.
Loaded symbols for /boot/kernel.old/crypto.ko.symbols
Reading symbols from /boot/kernel.old/cryptodev.ko.symbols...done.
Loaded symbols for /boot/kernel.old/cryptodev.ko.symbols
Reading symbols from /boot/kernel.old/dtraceall.ko.symbols...done.
Loaded symbols for /boot/kernel.old/dtraceall.ko.symbols
Reading symbols from /boot/kernel.old/profile.ko.symbols...done.
Loaded symbols for /boot/kernel.old/profile.ko.symbols
Reading symbols from /boot/kernel.old/cyclic.ko.symbols...done.
Loaded symbols for /boot/kernel.old/cyclic.ko.symbols
Reading symbols from /boot/kernel.old/dtrace.ko.symbols...done.
Loaded symbols for /boot/kernel.old/dtrace.ko.symbols
Reading symbols from /boot/kernel.old/systrace_freebsd32.ko.symbols...done.
Loaded symbols for /boot/kernel.old/systrace_freebsd32.ko.symbols
Reading symbols from /boot/kernel.old/systrace.ko.symbols...done.
Loaded symbols for /boot/kernel.old/systrace.ko.symbols
Reading symbols from /boot/kernel.old/sdt.ko.symbols...done.
Loaded symbols for /boot/kernel.old/sdt.ko.symbols
Reading symbols from /boot/kernel.old/lockstat.ko.symbols...done.
Loaded symbols for /boot/kernel.old/lockstat.ko.symbols
Reading symbols from /boot/kernel.old/fasttrap.ko.symbols...done.
Loaded symbols for /boot/kernel.old/fasttrap.ko.symbols
Reading symbols from /boot/kernel.old/fbt.ko.symbols...done.
Loaded symbols for /boot/kernel.old/fbt.ko.symbols
Reading symbols from /boot/kernel.old/dtnfscl.ko.symbols...done.
Loaded symbols for /boot/kernel.old/dtnfscl.ko.symbols
Reading symbols from /boot/kernel.old/dtmalloc.ko.symbols...done.
Loaded symbols for /boot/kernel.old/dtmalloc.ko.symbols
Reading symbols from /boot/kernel.old/dtio.ko.symbols...done.
Loaded symbols for /boot/kernel.old/dtio.ko.symbols
Reading symbols from /boot/modules/vboxdrv.ko...done.
Loaded symbols for /boot/modules/vboxdrv.ko
Reading symbols from /boot/modules/nvidia.ko...done.
Loaded symbols for /boot/modules/nvidia.ko
Reading symbols from /boot/kernel.old/fdescfs.ko.symbols...done.
Loaded symbols for /boot/kernel.old/fdescfs.ko.symbols
Reading symbols from /boot/kernel.old/uhid.ko.symbols...done.
Loaded symbols for /boot/kernel.old/uhid.ko.symbols
Reading symbols from /boot/kernel.old/ums.ko.symbols...done.
Loaded symbols for /boot/kernel.old/ums.ko.symbols
Reading symbols from /boot/modules/vboxnetflt.ko...done.
Loaded symbols for /boot/modules/vboxnetflt.ko
Reading symbols from /boot/kernel.old/netgraph.ko.symbols...done.
Loaded symbols for /boot/kernel.old/netgraph.ko.symbols
Reading symbols from /boot/kernel.old/ng_ether.ko.symbols...done.
Loaded symbols for /boot/kernel.old/ng_ether.ko.symbols
Reading symbols from /boot/modules/vboxnetadp.ko...done.
Loaded symbols for /boot/modules/vboxnetadp.ko
#0  doadump (textdump=<value optimized out>) at pcpu.h:236
236	pcpu.h: No such file or directory.
 	in pcpu.h
(kgdb) #0  doadump (textdump=<value optimized out>) at pcpu.h:236
#1  0xffffffff8051d6f0 in kern_reboot (howto=260)
     at /usr/src/sys/kern/kern_shutdown.c:447
#2  0xffffffff8051da77 in panic (fmt=<value optimized out>)
     at /usr/src/sys/kern/kern_shutdown.c:754
#3  0xffffffff80780d9a in trap_fatal (frame=<value optimized out>,
     eva=<value optimized out>) at /usr/src/sys/amd64/amd64/trap.c:873
#4  0xffffffff80780a45 in trap (frame=<value optimized out>)
     at /usr/src/sys/amd64/amd64/trap.c:224
#5  0xffffffff8076ad02 in calltrap ()
     at /usr/src/sys/amd64/amd64/exception.S:232
#6  0xffffffff80568be0 in vmem_free (vm=0xffffffff80bd7e00,
     addr=18446743528448167936, size=<value optimized out>)
     at /usr/src/sys/kern/subr_vmem.c:1197
#7  0xffffffff806f452c in uma_large_free (slab=0xfffffe02a3f0c000)
     at /usr/src/sys/vm/uma_core.c:1097
#8  0xffffffff805067f1 in free (addr=<value optimized out>,
     mtp=0xffffffff810f6c40) at /usr/src/sys/kern/kern_malloc.c:600
#9  0xffffffff80ed4f8f in arc_buf_destroy (buf=0xfffffe02a3ef4b40,
     recycle=<value optimized out>, all=0)
     at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c:1633
#10 0xffffffff80ed7cf3 in arc_buf_evict (buf=0xfffffe02a3ef4b40)
     at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c:3466
#11 0xffffffff80edd708 in dbuf_clear (db=0xfffffe02a3ef7c40)
     at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dbuf.c:1578
#12 0xffffffff80ef46e2 in dnode_evict_dbufs (dn=0xfffffe02a3f056b0)
     at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dnode_sync.c:405
#13 0xffffffff80eec227 in dmu_objset_evict_dbufs (os=<value optimized out>)
     at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_objset.c:588
#14 0xffffffff80f6693e in zfsvfs_teardown (zfsvfs=0xfffffe0020164000,
     unmounting=1)
     at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vfsops.c:1930
#15 0xffffffff80f68240 in zfs_umount (vfsp=0xfffffe001d162990,
     fflag=<value optimized out>)
     at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vfsops.c:2024
#16 0xffffffff805aed74 in dounmount (mp=0xfffffe001d162990, flags=524288,
     td=0xfffffe000d163000) at /usr/src/sys/kern/vfs_mount.c:1316
#17 0xffffffff805b6f81 in vfs_unmountall ()
     at /usr/src/sys/kern/vfs_subr.c:3392
#18 0xffffffff8051d494 in kern_reboot (howto=0)
     at /usr/src/sys/kern/kern_shutdown.c:430
#19 0xffffffff8051cfb0 in sys_reboot (td=<value optimized out>,
     uap=<value optimized out>) at /usr/src/sys/kern/kern_shutdown.c:191
#20 0xffffffff80781697 in amd64_syscall (td=0xfffffe000d163000, traced=0)
     at subr_syscall.c:134
#21 0xffffffff8076afeb in Xfast_syscall ()
     at /usr/src/sys/amd64/amd64/exception.S:391
#22 0x000000000040ec7c in ?? ()
Previous frame inner to this frame (corrupt stack?)
Current language:  auto; currently minimal
(kgdb)

------------------------------------------------------------------------
ps -axl

UID  PID PPID CPU PRI NI  VSZ RSS MWCHAN   STAT TT       TIME COMMAND
   0    0    0   0  -8  0    0   0 -        DLs   -    7:42.05 [kernel]
   0    1    0   0 -60  0 9428   0 -        RLs   -    0:00.37 [init]
   0    2    0   0 -16  0    0   0 ftcl     DL    -    0:00.00 [ftcleanup]
   0    3    0   0 -16  0    0   0 crypto_w DL    -    0:00.00 [crypto]
   0    4    0   0 -16  0    0   0 crypto_r DL    -    0:00.00 [crypto returns]
   0    5    0   0 -16  0    0   0 -        DL    -    0:00.02 [fdc0]
   0    6    0   0  -8  0    0   0 tx->tx_s DL    -    0:27.16 [zfskern]
   0    7    0   0 -16  0    0   0 waiting_ DL    -    0:00.00 [sctp_iterator]
   0    8    0   0 -16  0    0   0 ccb_scan DL    -    0:00.00 [xpt_thrd]
   0    9    0   0 -16  0    0   0 psleep   DL    -    0:00.01 [pagedaemon]
   0   10    0   0 -16  0    0   0 audit_wo DL    -    0:00.00 [audit]
   0   11    0   0 155  0    0   0 -        RL    -  190:06.94 [idle]
   0   12    0   0 -84  0    0   0 -        WL    -    0:23.59 [intr]
   0   13    0   0  -8  0    0   0 -        DL    -    0:21.46 [geom]
   0   14    0   0 -16  0    0   0 -        DL    -    0:00.83 [yarrow]
   0   15    0   0 -68  0    0   0 -        DL    -    0:02.21 [usb]
   0   16    0   0 -20  0    0   0 VBoxIS   DL    -    0:00.00 [TIMER]
   0   17    0   0 -16  0    0   0 psleep   DL    -    0:00.00 [vmdaemon]
   0   18    0   0 155  0    0   0 pgzero   DL    -    0:00.00 [pagezero]
   0   19    0   0 -16  0    0   0 kpsusp   DL    -    0:00.03 [bufdaemon]
   0   20    0   0  16  0    0   0 kpsusp   DL    -    0:07.87 [syncer]
   0   21    0   0 -16  0    0   0 kpsusp   DL    -    0:00.02 [vnlru]
   0 1192    0   0 -16  0    0   0 sleep    DL    -    0:00.00 [ng_queue]

------------------------------------------------------------------------
vmstat -s

  14427194 cpu context switches
   1380667 device interrupts
    337562 software interrupts
  52566588 traps
  64464504 system calls
        22 kernel threads created
     26404  fork() calls
     84967 vfork() calls
        27 rfork() calls
         0 swap pager pageins
         0 swap pager pages paged in
         0 swap pager pageouts
         0 swap pager pages paged out
     49778 vnode pager pageins
    370965 vnode pager pages paged in
       874 vnode pager pageouts
      1765 vnode pager pages paged out
         0 page daemon wakeups
         0 pages examined by the page daemon
      8322 pages reactivated
   2305973 copy-on-write faults
       596 copy-on-write optimized faults
  45375336 zero fill pages zeroed
         0 zero fill pages prezeroed
        62 intransit blocking page faults
  48421424 total VM faults taken
     49445 page faults requiring I/O
         0 pages affected by kernel thread creation
   2033792 pages affected by  fork()
   4458164 pages affected by vfork()
    190359 pages affected by rfork()
         0 pages cached
  66033440 pages freed
         0 pages freed by daemon
         0 pages freed by exiting processes
      1925 pages active
      7617 pages inactive
         4 pages in VM cache
   1564322 pages wired down
  14505711 pages free
      4096 bytes per page
  97717876 total name lookups
           cache hits (91% pos + 6% neg) system 0% per-directory
           deletions 0%, falsehits 0%, toolong 0%

------------------------------------------------------------------------
vmstat -m

          Type InUse MemUse HighUse Requests  Size(s)
       kdtrace   434   105K       -   112780  64,256
          kenv    84    11K       -      112  16,32,64,128
        kqueue     0     0K       -       78  256,512,2048
     proc-args     1     1K       -    84631  16,32,64,128,256
        DEVFSP     0     0K       -       38  64
         hhook     2     1K       -        2  256
       ithread   122    22K       -      122  32,128,256
        KTRACE   100    13K       -      100  128
        linker   401   197K       -      544  16,32,64,128,256,512,1024,2048,4096
         lockf     2     1K       -      196  64,128
    loginclass     1     1K       -       25  64
         cache     1     1K       -        1  32
        devbuf 17637 36004K       -    19791  16,32,64,128,256,512,1024,2048,4096
          temp    56    34K       -   302980  16,32,64,128,256,512,1024,2048,4096
        ip6ndp     5     1K       -        6  64,128
        module   301    38K       -      301  128
      mtx_pool     2    16K       -        2
           osd     4     1K       -       20  16,32,64,128
      pmchooks     1     1K       -        1  128
        NFS fh     0     0K       -        7  16
          pgrp     1     1K       -      215  128
       session     1     1K       -      176  128
          proc     2   256K       -        2
       subproc   174   168K       -   111554  512,4096
          cred    38     6K       -  1306976  64,256
        plimit     1     1K       -      461  256
       uidinfo     3    33K       -       41  128
        sysctl     0     0K       -     5933  16,32,64
     sysctloid  6788   337K       -     6976  16,32,64,128
     sysctltmp     0     0K       -    79008  16,32,64,128,256,4096
       tidhash     1   256K       -        1
       callout     9  3208K       -        9
          umtx   954   120K       -      954  128
      p1003.1b     1     1K       -        1  16
          SWAP     0     0K       -       12  64
           bus   947    91K       -     5406  16,32,64,128,256,512,1024
        bus-sc   141   293K       -     2336  16,32,64,128,256,512,1024,2048,4096
       devstat    34    69K       -       34  32,4096
  eventhandler    90     8K       -       90  64,128
          kobj   180   720K       -     1010  4096
       Per-cpu     1     1K       -        1  32
          rman   360    41K       -      797  16,32,128
          sbuf     0     0K       -     3735  16,32,64,128,256,512,1024,2048,4096
        sglist     3     1K       -        3  32
         stack     0     0K       -        2  256
     taskqueue    75    12K       -      139  16,32,64,256,1024
        Unitno    20     2K       -    18582  32,64
          vmem     2  1152K       -        9
      ioctlops     0     0K       -   169348  16,32,64,128,256,512,1024,2048
        select    87    11K       -       87  128
           iov     0     0K       -    36910  16,64,128,256,512
           msg     4    30K       -        4  2048,4096
           sem     4   106K       -        4  2048,4096
           shm     1    20K       -      122  2048
           tty    19    19K       -       22  1024,2048
           pts     0     0K       -        1  256
      mbuf_tag     0     0K       -      728  32,128
         shmfd     1     8K       -        1
        soname     0     0K       -    23565  16,32,128
           pcb    14  8341K       -      113  16,32,128,1024,2048
     newnfsmnt     0     0K       -        1  1024
           acl     0     0K       -       32  4096
      vfscache     1 16384K       -        1
      vfs_hash     1  8192K       -        1
        vnodes     1     1K       -        1  256
     pfs_nodes    21     6K       -       21  256
   pfs_vncache    25     2K       -     1055  64
         mount   164     9K       -     1854  16,32,64,128,256,512
          GEOM   338    59K       -     3552  16,32,64,128,256,512,1024,2048
   vnodemarker     0     0K       -     6559  512
           BPF     3     1K       -        3  128
        acpica  1830   187K       -    52091  16,32,64,128,256,512,1024,2048
         ifnet     4     7K       -        4  128,2048
        ifaddr    48    15K       -       48  32,64,128,256,512,2048,4096
   ether_multi    40     3K       -       54  16,32,64
         clone     6     1K       -        6  128
        arpcom     2     1K       -        2  16
       lltable    13     5K       -       13  256,512
          UART     6     5K       -        6  16,1024
       CAM CCB    13    26K       -      119  2048
      acpitask     1     8K       -        1
       acpisem    22     3K       -       22  128
      CAM path    22     1K       -       68  32
       acpidev    36     3K       -       36  64
      routetbl    35     5K       -     1338  32,64,128,256,512
          igmp     3     1K       -        3  256
      in_multi     2     1K       -        2  256
    CAM periph    16     4K       -       40  16,32,64,128,256
     CAM queue    31     8K       -      100  16,32,512
CAM dev queue     8     1K       -        8  32
     sctp_a_it     0     0K       -        3  16
      sctp_vrf     1     1K       -        1  64
      sctp_ifa     5     1K       -        5  128
      sctp_ifn     2     1K       -        2  128
     sctp_iter     0     0K       -        3  256
     hostcache     1    28K       -        1
      syncache     1    64K       -        1
   in6_mfilter     0     0K       -        1  1024
     in6_multi    25     3K       -       27  32,256
  ip6_moptions     0     0K       -        2  32,256
     raid_data     0     0K       -      552  32,128,256
           mld     3     1K       -        3  128
       CAM SIM     8     2K       -        8  256
       CAM XPT    58     4K       -      354  16,32,64,128,256,1024
       NFS FHA     1     2K       -        1  2048
           rpc     8     3K       -       34  32,64,128,256,512,1024
audit_evclass   188     6K       -      229  32
     vm_pgdata     1  8192K       -        7  128
       UMAHash     3    73K       -       14  512,1024,2048,4096
       scsi_cd     0     0K       -       11  16
           USB    48    86K       -       56  16,32,64,128,256,512,1024,4096
        USBdev    71    20K       -      112  32,64,128,512,2048,4096
      pci_link    16     2K       -       16  32,64,128
     acpi_perf     8     1K       -        8  64
       memdesc     1     4K       -        1  4096
      atkbddev     2     1K       -        2  64
md_nvidia_data     0     0K       -       90  512
        kbdmux     7    18K       -        7  16,512,1024,2048
           LED     4     1K       -        4  16,128
   md_sii_data     0     0K       -       90  512
       CAM DEV    15    30K       -       24  2048
       ata_pci     1     1K       -        1  64
      acpiintr     1     1K       -        1  64
      ppbusdev     2     1K       -        2  256
        apmdev     1     1K       -        1  128
    madt_table     0     0K       -        1  4096
        isadev     5     1K       -        5  128
random_adaptors     1     1K       -        1  32
       entropy  1024    64K       -     1024  64
        DEVFS3   223    56K       -      269  256
       io_apic     2     4K       -        2  2048
        DEVFS1   194    97K       -      236  512
           MCA     8     1K       -        8  128
          cdev     8     2K       -        8  256
           msi     2     1K       -        2  128
      nexusdev     4     1K       -        4  16
         DEVFS    42     1K       -       43  16,128
      filedesc    27    47K       -   111424  16,32,64,128,2048,4096
filedesc_to_leader     0     0K       -       36  64
         sigio     0     0K       -        1  64
      filecaps     0     0K       -        2  64
         linux    16     1K       -       73  32,64
         mixer     1     4K       -        1  4096
        feeder    16     2K       -       18  32,128
    kstat_data     5     1K       -        5  64
       solaris 433591 1104029K       - 92658736  16,32,64,128,256,512,1024,2048,4096
         futex     0     0K       -      304  128
        spicds     7     1K       -        7  128
      envy24ht    15   195K       -       15  64,2048
        cpuctl     1     1K       -        9  64,4096
        crypto     1     1K       -        1  512
         xform     0     0K       -      167  16,32
        cyclic    32     3K       -       32  16,64,128
      futex wp     0     0K       -      154  32
           fbt     1   256K       -        1
      iprtheap    23    56K       -       23  32,64,128,256,2048
        nvidia   179   862K       -      182  16,32,64,128,256,512,1024,2048,4096
   fdesc_mount     1     1K       -        1  16
netgraph_node     4     1K       -        4  128,256
      netgraph     2     1K       -        2  64

------------------------------------------------------------------------
vmstat -z

ITEM                   SIZE  LIMIT     USED     FREE      REQ FAIL SLEEP

UMA Kegs:               384,      0,     172,       8,     172,   0,   0
UMA Zones:             1664,      0,     188,       0,     188,   0,   0
UMA Slabs:               80,      0,  915068,   53332, 2022489,   0,   0
UMA RCntSlabs:           88,      0,     955,      35,     955,   0,   0
UMA Hash:               256,      0,       7,       8,      10,   0,   0
4 Bucket:                32,      0,     665,     835,   73772,33766,   0
8 Bucket:                64,      0,     496,     930,   39091,32324,   0
16 Bucket:              128,      0,   12763,     877,   49961,13508,   0
32 Bucket:              256,      0,    8095,     650,   44638, 101,   0
64 Bucket:              512,      0,   12112,     216,  124179,5726,   0
128 Bucket:            1024,      0,   36359,     193,  363238,3321,   0
vmem btag:               56,      0,   47826,   63147,  135656,1564,   0
VM OBJECT:              256,      0,   14484,   53661, 1807265,   0,   0
RADIX NODE:             144,      0,   15125,  149224, 5870448,  39,   0
MAP:                    240,      0,       3,      61,       3,   0,   0
KMAP ENTRY:             128,      0,      13,     638,      13,   2,   0
MAP ENTRY:              128,      0,       6,    3249, 4017261,   0,   0
VMSPACE:                408,      0,       1,     377,  111390,   0,   0
fakepg:                 104,      0,       0,       0,       0,   0,   0
mt_zone:               4112,      0,     288,       0,     288,   0,   0
16:                      16,      0,       5,     246,       5,   0,   0
16:                      16,      0,    2467,     796,    2628,   0,   0
16:                      16,      0,       2,     249,       2,   0,   0
16:                      16,      0,      85,     919,   62760,   0,   0
16:                      16,      0,       5,     999,    5322,   0,   0
16:                      16,      0,     494,    1012,   64010,   0,   0
16:                      16,      0,      47,     706,      82,   0,   0
16:                      16,      0,   17593,   63982, 4986289,   0,   0
32:                      32,      0,       0,     125,       2,   0,   0
32:                      32,      0,    2632,     868,    2747,   0,   0
32:                      32,      0,      10,     240,      10,   0,   0
32:                      32,      0,      86,    1289,   67620,   0,   0
32:                      32,      0,     250,     875,    1137,   0,   0
32:                      32,      0,     324,    1176,    3754,   0,   0
32:                      32,      0,      68,     932,   18653,   0,   0
32:                      32,      0,    4725,  147525, 6677520,   0,   0
64:                      64,      0,       3,     555,       9,   0,   0
64:                      64,      0,     319,     735,     346,   0,   0
64:                      64,      0,      14,     296,      14,   0,   0
64:                      64,      0,     761,    1037,   66835,   0,   0
64:                      64,      0,      42,    1260,  111464,   0,   0
64:                      64,      0,    8997,    1047,  794831,   0,   0
64:                      64,      0,    1076,     846,    1112,   0,   0
64:                      64,      0,  197922,  540312,33558786,   0,   0
128:                    128,      0,       5,     522,       9,   0,   0
128:                    128,      0,    2003,     880,    2538,   0,   0
128:                    128,      0,       6,    1017,     220,   0,   0
128:                    128,      0,    1257,     882,   22917,   0,   0
128:                    128,      0,    1319,     448,    1723,   0,   0
128:                    128,      0,     963,    1858,   46163,   0,   0
128:                    128,      0,       3,     276,       3,   0,   0
128:                    128,      0,   12017,  123143,15266248,   0,   0
256:                    256,      0,       2,     133,       3,   0,   0
256:                    256,      0,      12,     483,     498,   0,   0
256:                    256,      0,       6,     129,       6,   0,   0
256:                    256,      0,     241,     404,     902,   0,   0
256:                    256,      0,     495,     390,    1814,   0,   0
256:                    256,      0,     327,     453,  655716,   0,   0
256:                    256,      0,       0,      75,       1,   0,   0
256:                    256,      0,   15088,  662687,15988736,   0,   0
512:                    512,      0,       0,      88,      13,   0,   0
512:                    512,      0,      39,     169,     219,   0,   0
512:                    512,      0,       0,     232,    6559,   0,   0
512:                    512,      0,     235,     189,    1448,   0,   0
512:                    512,      0,      18,     102,      18,   0,   0
512:                    512,      0,     369,     127,   12372,   0,   0
512:                    512,      0,       0,       0,       0,   0,   0
512:                    512,      0,   18589,  356091, 8996036,   0,   0
1024:                  1024,      0,       2,      26,       2,   0,   0
1024:                  1024,      0,       1,      51,     100,   0,   0
1024:                  1024,      0,       4,      24,       5,   0,   0
1024:                  1024,      0,       6,      74,    1720,   0,   0
1024:                  1024,      0,      19,      21,      20,   0,   0
1024:                  1024,      0,      25,     263,  135383,   0,   0
1024:                  1024,      0,       0,      16,       1,   0,   0
1024:                  1024,      0,     368,    3712,   76117,   0,   0
2048:                  2048,      0,       0,      46,     121,   0,   0
2048:                  2048,      0,      32,      36,     147,   0,   0
2048:                  2048,      0,       1,       5,       1,   0,   0
2048:                  2048,      0,      15,      15,     281,   0,   0
2048:                  2048,      0,      24,     132,  111405,   0,   0
2048:                  2048,      0,      23,      41,     211,   0,   0
2048:                  2048,      0,       5,       1,       5,   0,   0
2048:                  2048,      0,     634,   19428,  141789,   0,   0
4096:                  4096,      0,       1,       0,       1,   0,   0
4096:                  4096,      0,       2,       0,       2,   0,   0
4096:                  4096,      0,     180,       5,    1010,   0,   0
4096:                  4096,      0,      12,       2,      46,   0,   0
4096:                  4096,      0,       1,      11,      16,   0,   0
4096:                  4096,      0,      78,      74,  115621,   0,   0
4096:                  4096,      0,       0,       0,       0,   0,   0
4096:                  4096,      0,  156695,  678095, 5998395,   0,   0
uint64 pcpu:              8,      0,    1386,     150,    1386,   0,   0
SLEEPQUEUE:              80,      0,     478,     669,     478,   0,   0
Files:                   80,      0,       0,    1650, 7096539,   0,   0
TURNSTILE:              136,      0,     478,     202,     478,   0,   0
rl_entry:                40,      0,     115,     885,     115,   0,   0
umtx pi:                 96,      0,       0,       0,       0,   0,   0
MAC labels:              40,      0,       0,       0,       0,   0,   0
PROC:                  1208,      0,      22,     128,  111420,   0,   0
THREAD:                1168,      0,     410,      67,    1358,   0,   0
cpuset:                  72,      0,     283,     597,     438,   0,   0
cyclic_id_cache:         64,      0,       0,       0,       0,   0,   0
audit_record:          1240,      0,       0,       0,       0,   0,   0
mbuf_packet:            256, 25727505,    1023,     872,   55326,   7,   0
mbuf:                   256, 25727505,       1,    1012,  188962,   0,   0
mbuf_cluster:          2048, 4019922,    1882,       4,    1882, 295,   0
mbuf_jumbo_page:       4096, 2009961,       0,      12,     121,   1,   0
mbuf_jumbo_9k:         9216, 1786632,       0,       0,       0,   0,   0
mbuf_jumbo_16k:       16384, 1339972,       0,       0,       0,   0,   0
mbuf_ext_refcnt:          4,      0,       0,       0,       0,   0,   0
dtrace_state_cache:    4096,      0,       0,       0,       0,   0,   0
ttyinq:                 160,      0,       0,     500,     570,   0,   0
ttyoutq:                256,      0,       0,     495,     303,   0,   0
g_bio:                  248,      0,       0,     800, 3542613,   0,   0
ata_request:            336,      0,       0,     209,    1256,   0,   0
vtnet_tx_hdr:            24,      0,       0,       0,       0,   0,   0
cryptop:                 88,      0,       0,       0,       0,   0,   0
cryptodesc:              72,      0,       0,       0,       0,   0,   0
nv_stack_t:           12288,      0,       2,       2,      12,   0,   0
FPU_save_area:          512,      0,       0,       0,       0,   0,   0
taskq_zone:              48,      0,       0,       0,       0,   0,   0
taskq_zone:              48,      0,       0,    1660,  152942,   0,   0
VNODE:                  472,      0,   17005,  114979,  164557,   0,   0
VNODEPOLL:              112,      0,       0,     140,       1,   0,   0
BUF TRIE:               144,      0,       0,  105948,       0,   0,   0
NAMEI:                 1024,      0,       0,     132,15076677,   0,   0
S VFS Cache:            108,      0,   16355,  114895, 1404872,   0,   0
STS VFS Cache:          148,      0,       0,       0,       0,   0,   0
L VFS Cache:            328,      0,     144,     756,    3759,   0,   0
LTS VFS Cache:          368,      0,       0,       0,       0,   0,   0
NCLNODE:                528,      0,       0,      14,       1,   0,   0
space_seg_cache:         64,      0,  135051,  445331, 3229423,   0,   0
zio_cache:              944,      0,       1,  939411,26694286,   0,   0
zio_link_cache:          48,      0,       0,  941137,24475658,   0,   0
sa_cache:                80,      0,   16782,  115218,  163215,   0,   0
dnode_t:                856,      0,   19112,  122432,  166451,   0,   0
dmu_buf_impl_t:         224,      0,  178914,  213208,  806869,   0,   0
arc_buf_hdr_t:          216,      0,  292380,     372,  470888,   0,   0
arc_buf_t:               72,      0,  168353,  112312,  669851,   0,   0
zil_lwb_cache:          192,      0,       2,     678,    1105,   0,   0
zfs_znode_cache:        368,      0,   16782,  114878,  163215,   0,   0
Mountpoints:            816,      0,      13,      52,      30,   0,   0
pipe:                   744,      0,       0,     145,    8692,   0,   0
ksiginfo:               112,      0,     139,    1751,  278167,   0,   0
itimer:                 352,      0,       0,      33,       1,   0,   0
KNOTE:                  128,      0,       0,     775,   44989,   0,   0
socket:                 680, 2063352,       1,     131,    4149,   0,   0
unpcb:                  240, 2063360,       0,     496,    3740,   0,   0
ipq:                     56, 125670,       0,       0,       0,   0,   0
udp_inpcb:              392, 2063360,       0,     240,     366,   0,   0
udpcb:                   16, 2063471,       0,    1004,     366,   0,   0
tcp_inpcb:              392, 2063360,       2,     238,      37,   0,   0
tcpcb:                 1024, 2063352,       1,     103,      37,   0,   0
tcptw:                   72,  27775,       1,     164,       3,   0,   0
syncache:               152,  15366,       0,      78,       2,   0,   0
hostcache:              136,  15370,       3,      84,       3,   0,   0
tcpreass:                40, 251300,       0,       0,       0,   0,   0
sackhole:                32,      0,       0,     250,      71,   0,   0
sctp_ep:               1408, 2063352,       0,       0,       0,   0,   0
sctp_asoc:             2344,  40000,       0,       0,       0,   0,   0
sctp_laddr:              48,  80012,       0,     415,       4,   0,   0
sctp_raddr:             728,  80000,       0,       0,       0,   0,   0
sctp_chunk:             136, 400026,       0,       0,       0,   0,   0
sctp_readq:             104, 400026,       0,       0,       0,   0,   0
sctp_stream_msg_out:    104, 400026,       0,       0,       0,   0,   0
sctp_asconf:             40, 400000,       0,       0,       0,   0,   0
sctp_asconf_ack:         48, 400060,       0,       0,       0,   0,   0
ripcb:                  392, 2063360,       0,       0,       0,   0,   0
rtentry:                200,      0,      17,     243,      17,   0,   0
selfd:                   56,      0,      40,    1238,  390516,   0,   0
SWAPMETA:               288, 8039850,       0,       0,       0,   0,   0
NetGraph items:          72,   4123,       0,       0,       0,   0,   0
NetGraph data items:     72,    527,       0,       0,       0,   0,   0


------------------------------------------------------------------------
vmstat -i

interrupt                          total       rate
irq1: atkbd0                           4          0
irq6: fdc0                            22          0
irq14: ata0                         2489          7
irq17: uhci0 ehci0                185135        562
irq19: uhci1 ahci0+              1085638       3299
cpu0:timer                       2661262       8088
irq256: em0                       107379        326
cpu7:timer                       2592585       7880
cpu6:timer                       2568730       7807
cpu4:timer                       2516493       7648
cpu1:timer                       2633814       8005
cpu2:timer                       2601955       7908
cpu3:timer                       2623175       7973
cpu5:timer                       2573700       7822
Total                           22152381      67332

------------------------------------------------------------------------
pstat -T

   0/2063348 files
0M/0M swap space

------------------------------------------------------------------------
pstat -s

Device          512-blocks     Used    Avail Capacity

------------------------------------------------------------------------
iostat

iostat: kvm_read(_tk_nin): invalid address (0x0)
iostat: disabling TTY statistics
             ada0             ada1             ada2             cpu
   KB/t tps  MB/s   KB/t tps  MB/s   KB/t tps  MB/s  us ni sy in id
  38.80 663 25.13  22.66 552 12.21  22.28 575 12.51   8 49  4  0 39

------------------------------------------------------------------------
ipcs -a

Message Queues:
T           ID          KEY MODE        OWNER    GROUP    CREATOR  CGROUP                 CBYTES                 QNUM               QBYTES        LSPID        LRPID STIME    RTIME    CTIME

Shared Memory:
T           ID          KEY MODE        OWNER    GROUP    CREATOR  CGROUP         NATTCH        SEGSZ         CPID         LPID ATIME    DTIME    CTIME

Semaphores:
T           ID          KEY MODE        OWNER    GROUP    CREATOR  CGROUP          NSEMS OTIME    CTIME


------------------------------------------------------------------------
ipcs -T

msginfo:
 	msgmax:        16384	(max characters in a message)
 	msgmni:           40	(# of message queues)
 	msgmnb:         2048	(max characters in a message queue)
 	msgtql:           40	(max # of messages in system)
 	msgssz:            8	(size of a message segment)
 	msgseg:         2048	(# of message segments in system)

shminfo:
 	shmmax:    536870912	(max shared memory segment size)
 	shmmin:            1	(min shared memory segment size)
 	shmmni:          192	(max number of shared memory identifiers)
 	shmseg:          128	(max shared memory segments per process)
 	shmall:       131072	(max amount of shared memory in pages)

seminfo:
 	semmni:           50	(# of semaphore identifiers)
 	semmns:          340	(# of semaphores in system)
 	semmnu:          150	(# of undo structures in system)
 	semmsl:          340	(max # of semaphores per id)
 	semopm:          100	(max # of operations per semop call)
 	semume:           50	(max # of undo entries per process)
 	semusz:          632	(size in bytes of undo structure)
 	semvmx:        32767	(semaphore maximum value)
 	semaem:        16384	(adjust on exit max value)


------------------------------------------------------------------------
nfsstat

Client Info:
Rpc Counts:
   Getattr   Setattr    Lookup  Readlink      Read     Write    Create    Remove
         2         0         0         0         0         0         0         0
    Rename      Link   Symlink     Mkdir     Rmdir   Readdir  RdirPlus    Access
         0         0         0         0         0         0         0         0
     Mknod    Fsstat    Fsinfo  PathConf    Commit
         0         1         1         0         0
Rpc Info:
  TimedOut   Invalid X Replies   Retries  Requests
         0         0         0         0         4
Cache Info:
Attr Hits    Misses Lkup Hits    Misses BioR Hits    Misses BioW Hits    Misses
         0         0         0         0         0         0         0         0
BioRLHits    Misses BioD Hits    Misses DirE Hits    Misses Accs Hits    Misses
         0         0         0         0         0         0         0         0

Server Info:
   Getattr   Setattr    Lookup  Readlink      Read     Write    Create    Remove
         0         0         0         0         0         0         0         0
    Rename      Link   Symlink     Mkdir     Rmdir   Readdir  RdirPlus    Access
         0         0         0         0         0         0         0         0
     Mknod    Fsstat    Fsinfo  PathConf    Commit
         0         0         0         0         0
Server Ret-Failed
                 0
Server Faults
             0
Server Cache Stats:
    Inprog      Idem  Non-idem    Misses
         0         0         0         0
Server Write Gathering:
  WriteOps  WriteRPC   Opsaved
         0         0         0

------------------------------------------------------------------------
netstat -s

tcp:
 	66097 packets sent
 		65690 data packets (15870623 bytes)
 		122 data packets (134444 bytes) retransmitted
 		16 data packets unnecessarily retransmitted
 		0 resends initiated by MTU discovery
 		272 ack-only packets (143 delayed)
 		0 URG only packets
 		0 window probe packets
 		0 window update packets
 		13 control packets
 	35284 packets received
 		34167 acks (for 15880648 bytes)
 		210 duplicate acks
 		0 acks for unsent data
 		1373 packets (98989 bytes) received in-sequence
 		0 completely duplicate packets (0 bytes)
 		0 old duplicate packets
 		0 packets with some dup. data (0 bytes duped)
 		0 out-of-order packets (0 bytes)
 		0 packets (0 bytes) of data after window
 		0 window probes
 		0 window update packets
 		0 packets received after close
 		0 discarded for bad checksums
 		0 discarded for bad header offset fields
 		0 discarded because packet too short
 		0 discarded due to memory problems
 	6 connection requests
 	1 connection accept
 	0 bad connection attempts
 	0 listen queue overflows
 	0 ignored RSTs in the windows
 	7 connections established (including accepts)
 	35 connections closed (including 0 drops)
 		3 connections updated cached RTT on close
 		3 connections updated cached RTT variance on close
 		1 connection updated cached ssthresh on close
 	0 embryonic connections dropped
 	31136 segments updated rtt (of 31212 attempts)
 	48 retransmit timeouts
 		0 connections dropped by rexmit timeout
 	0 persist timeouts
 		0 connections dropped by persist timeout
 	0 Connections (fin_wait_2) dropped because of timeout
 	0 keepalive timeouts
 		0 keepalive probes sent
 		0 connections dropped by keepalive
 	2108 correct ACK header predictions
 	902 correct data packet header predictions
 	2 syncache entries added
 		0 retransmitted
 		0 dupsyn
 		0 dropped
 		1 completed
 		0 bucket overflow
 		0 cache overflow
 		1 reset
 		0 stale
 		0 aborted
 		0 badack
 		0 unreach
 		0 zone failures
 	2 cookies sent
 	0 cookies received
 	3 hostcache entries added
 		0 bucket overflow
 	32 SACK recovery episodes
 	55 segment rexmits in SACK recovery episodes
 	45072 byte rexmits in SACK recovery episodes
 	269 SACK options (SACK blocks) received
 	0 SACK options (SACK blocks) sent
 	0 SACK scoreboard overflow
 	0 packets with ECN CE bit set
 	0 packets with ECN ECT(0) bit set
 	0 packets with ECN ECT(1) bit set
 	0 successful ECN handshakes
 	0 times ECN reduced the congestion window
udp:
 	3446 datagrams received
 	0 with incomplete header
 	0 with bad data length field
 	0 with bad checksum
 	0 with no checksum
 	0 dropped due to no socket
 	2429 broadcast/multicast datagrams undelivered
 	0 dropped due to full socket buffers
 	0 not for hashed pcb
 	1017 delivered
 	1018 datagrams output
 	0 times multicast source filter matched
ip:
 	38131 total packets received
 	0 bad header checksums
 	0 with size smaller than minimum
 	0 with data size < data length
 	0 with ip length > max ip packet size
 	0 with header length < data size
 	0 with data length < header length
 	0 with bad options
 	0 with incorrect version number
 	0 fragments received
 	0 fragments dropped (dup or out of space)
 	0 fragments dropped after timeout
 	0 packets reassembled ok
 	38011 packets for this host
 	120 packets for unknown/unsupported protocol
 	0 packets forwarded (0 packets fast forwarded)
 	0 packets not forwardable
 	0 packets received for unknown multicast group
 	0 redirects sent
 	66397 packets sent from this host
 	0 packets sent with fabricated ip header
 	0 output packets dropped due to no bufs, etc.
 	0 output packets discarded due to no route
 	0 output datagrams fragmented
 	0 fragments created
 	0 datagrams that can't be fragmented
 	0 tunneling packets that can't find gif
 	0 datagrams with bad address in header
icmp:
 	0 calls to icmp_error
 	0 errors not generated in response to an icmp message
 	0 messages with bad code fields
 	0 messages less than the minimum length
 	0 messages with bad checksum
 	0 messages with bad length
 	0 multicast echo requests ignored
 	0 multicast timestamp requests ignored
 	0 message responses generated
 	0 invalid return addresses
 	0 no return routes
igmp:
 	120 messages received
 	0 messages received with too few bytes
 	0 messages received with wrong TTL
 	0 messages received with bad checksum
 	29 V1/V2 membership queries received
 	0 V3 membership queries received
 	0 membership queries received with invalid field(s)
 	29 general queries received
 	0 group queries received
 	0 group-source queries received
 	0 group-source queries dropped
 	27 membership reports received
 	0 membership reports received with invalid field(s)
 	0 membership reports received for groups to which we belong
 	0 V3 reports received without Router Alert
 	0 membership reports sent
arp:
 	3 ARP requests sent
 	92 ARP replies sent
 	2019 ARP requests received
 	2 ARP replies received
 	2021 ARP packets received
 	0 total packets dropped due to no ARP entry
 	0 ARP entrys timed out
 	0 Duplicate IPs seen
ip6:
 	719 total packets received
 	0 with size smaller than minimum
 	0 with data size < data length
 	0 with bad options
 	0 with incorrect version number
 	0 fragments received
 	0 fragments dropped (dup or out of space)
 	0 fragments dropped after timeout
 	0 fragments that exceeded limit
 	0 packets reassembled ok
 	719 packets for this host
 	0 packets forwarded
 	0 packets not forwardable
 	0 redirects sent
 	727 packets sent from this host
 	0 packets sent with fabricated ip header
 	0 output packets dropped due to no bufs, etc.
 	4 output packets discarded due to no route
 	0 output datagrams fragmented
 	0 fragments created
 	0 datagrams that can't be fragmented
 	0 packets that violated scope rules
 	0 multicast packets which we don't join
 	Input histogram:
 		UDP: 719
 	Mbuf statistics:
 		366 one mbuf
 		569 one ext mbuf
 		0 two or more ext mbuf
 	0 packets whose headers are not contiguous
 	0 tunneling packets that can't find gif
 	0 packets discarded because of too many headers
 	0 failures of source address selection
 	Source addresses selection rule applied:
icmp6:
 	0 calls to icmp6_error
 	0 errors not generated in response to an icmp6 message
 	0 errors not generated because of rate limitation
 	Output histogram:
 		neighbor solicitation: 1
 	0 messages with bad code fields
 	0 messages < minimum length
 	0 bad checksums
 	0 messages with bad length
 	Histogram of error messages to be generated:
 		0 no route
 		0 administratively prohibited
 		0 beyond scope
 		0 address unreachable
 		0 port unreachable
 		0 packet too big
 		0 time exceed transit
 		0 time exceed reassembly
 		0 erroneous header field
 		0 unrecognized next header
 		0 unrecognized option
 		0 redirect
 		0 unknown
 	0 message responses generated
 	0 messages with too many ND options
 	0 messages with bad ND options
 	0 bad neighbor solicitation messages
 	0 bad neighbor advertisement messages
 	0 bad router solicitation messages
 	0 bad router advertisement messages
 	0 bad redirect messages
 	0 path MTU changes
rip6:
 	0 messages received
 	0 checksum calculations on inbound
 	0 messages with bad checksum
 	0 messages dropped due to no socket
 	0 multicast messages dropped due to no socket
 	0 messages dropped due to full socket buffers
 	0 delivered
 	0 datagrams output

------------------------------------------------------------------------
netstat -m

1024/1884/2908 mbufs in use (current/cache/total)
1010/876/1886/4019922 mbuf clusters in use (current/cache/total/max)
1023/872 mbuf+clusters out of packet secondary zone in use (current/cache)
0/12/12/2009961 4k (page size) jumbo clusters in use (current/cache/total/max)
0/0/0/1786632 9k jumbo clusters in use (current/cache/total/max)
0/0/0/1339972 16k jumbo clusters in use (current/cache/total/max)
2276K/2271K/4547K bytes allocated to network (current/cache/total)
0/295/7 requests for mbufs denied (mbufs/clusters/mbuf+clusters)
0/0/0 requests for mbufs delayed (mbufs/clusters/mbuf+clusters)
0/0/0 requests for jumbo clusters delayed (4k/9k/16k)
1/0/0 requests for jumbo clusters denied (4k/9k/16k)
0 requests for sfbufs denied
0 requests for sfbufs delayed
0 requests for I/O initiated by sendfile

------------------------------------------------------------------------
netstat -id

Name    Mtu Network       Address              Ipkts Ierrs Idrop    Opkts Oerrs  Coll Drop
em0    1500 <Link#1>      00:30:48:f2:29:9c    40369     0     0    66492     0     0    0 
em0    1500 192.168.200.0 borg                 37204     -     -    66540     -     -    - 
em0    1500 fe80::230:48f fe80::230:48ff:fe        0     -     -        5     -     -    - 
em1*   1500 <Link#2>      00:30:48:f2:29:9d        0     0     0        0     0     0    0 
lo0   16384 <Link#3>                             719     0     0      719     0     0    0 
lo0   16384 localhost     ::1                    719     -     -      719     -     -    - 
lo0   16384 fe80::1%lo0   fe80::1                  0     -     -        0     -     -    - 
lo0   16384 your-net      localhost                0     -     -        0     -     -    -

------------------------------------------------------------------------
netstat -anr

Routing tables

Internet:
Destination        Gateway            Flags    Refs      Use  Netif Expire
default            192.168.200.11     UGS         0      229    em0
127.0.0.1          link#3             UH          0        0    lo0
192.168.200.0/24   link#1             U           0    66168    em0
192.168.200.4      link#1             UHS         0        0    lo0

Internet6:
Destination                       Gateway                       Flags      Netif Expire
::/96                             ::1                           UGRS        lo0
::1                               link#3                        UH          lo0
::ffff:0.0.0.0/96                 ::1                           UGRS        lo0
fe80::/10                         ::1                           UGRS        lo0
fe80::%em0/64                     link#1                        U           em0
fe80::230:48ff:fef2:299c%em0      link#1                        UHS         lo0
fe80::%lo0/64                     link#3                        U           lo0
fe80::1%lo0                       link#3                        UHS         lo0
ff01::%em0/32                     fe80::230:48ff:fef2:299c%em0  U           em0
ff01::%lo0/32                     ::1                           U           lo0
ff02::/16                         ::1                           UGRS        lo0
ff02::%em0/32                     fe80::230:48ff:fef2:299c%em0  U           em0
ff02::%lo0/32                     ::1                           U           lo0

------------------------------------------------------------------------
netstat -anA

Active Internet connections (including servers)
Tcpcb            Proto Recv-Q Send-Q Local Address      Foreign Address    (state)
fffffe0020d44360 tcp4       0      0 192.168.200.4.22   192.168.200.87.599 TIME_WAIT
fffffe02a3780c00 tcp4       0      0 192.168.200.4.4759 198.20.8.246.80    LAST_ACK

------------------------------------------------------------------------
netstat -aL

Current listen queue sizes (qlen/incqlen/maxqlen)
Proto Listen         Local Address

------------------------------------------------------------------------
fstat

fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
fstat: can't read znode_phys at 0x1
USER     CMD          PID   FD MOUNT      INUM MODE         SZ|DV R/W
root     ng_queue    1192 root -         -       error    -
root     ng_queue    1192   wd -         -       error    -
root     init           1 root -         -       error    -
root     init           1   wd -         -       error    -
root     init           1 text -         -       error    -
root     kernel         0 root -         -       error    -
root     kernel         0   wd -         -       error    -

------------------------------------------------------------------------
dmesg

Copyright (c) 1992-2013 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
 	The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 10.0-CURRENT #17: Sat Aug 10 14:10:47 CDT 2013
     root_at_borg.lerctr.org:/usr/obj/usr/src/sys/BORG-DTRACE amd64
FreeBSD clang version 3.3 (tags/RELEASE_33/final 183502) 20130610
CPU: Intel(R) Xeon(R) CPU           E5410  _at_ 2.33GHz (2327.54-MHz K8-class CPU)
   Origin = "GenuineIntel"  Id = 0x10676  Family = 0x6  Model = 0x17  Stepping = 6
   Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
   Features2=0xce3bd<SSE3,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,SSE4.1>
   AMD Features=0x20100800<SYSCALL,NX,LM>
   AMD Features2=0x1<LAHF>
   TSC: P-state invariant, performance statistics
real memory  = 68719476736 (65536 MB)
avail memory = 65660014592 (62618 MB)
Event timer "LAPIC" quality 400
ACPI APIC Table: <PTLTD  	 APIC  >
FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs
FreeBSD/SMP: 2 package(s) x 4 core(s)
  cpu0 (BSP): APIC ID:  0
  cpu1 (AP): APIC ID:  1
  cpu2 (AP): APIC ID:  2
  cpu3 (AP): APIC ID:  3
  cpu4 (AP): APIC ID:  4
  cpu5 (AP): APIC ID:  5
  cpu6 (AP): APIC ID:  6
  cpu7 (AP): APIC ID:  7
ioapic0 <Version 2.0> irqs 0-23 on motherboard
ioapic1 <Version 2.0> irqs 24-47 on motherboard
kbd1 at kbdmux0
netmap: loaded module
random: <Software, Yarrow> initialized
cryptosoft0: <software crypto> on motherboard
acpi0: <PTLTD   RSDT> on motherboard
acpi0: Power Button (fixed)
unknown: I/O range not supported
cpu0: <ACPI CPU> on acpi0
cpu1: <ACPI CPU> on acpi0
cpu2: <ACPI CPU> on acpi0
cpu3: <ACPI CPU> on acpi0
cpu4: <ACPI CPU> on acpi0
cpu5: <ACPI CPU> on acpi0
cpu6: <ACPI CPU> on acpi0
cpu7: <ACPI CPU> on acpi0
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff irq 0,8 on acpi0
Timecounter "HPET" frequency 14318180 Hz quality 950
Event timer "HPET" frequency 14318180 Hz quality 350
Event timer "HPET1" frequency 14318180 Hz quality 340
Event timer "HPET2" frequency 14318180 Hz quality 340
atrtc0: <AT realtime clock> port 0x70-0x71 on acpi0
Event timer "RTC" frequency 32768 Hz quality 0
attimer0: <AT timer> port 0x40-0x43,0x50-0x53 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> at device 2.0 on pci0
pci1: <ACPI PCI bus> on pcib1
pcib2: <ACPI PCI-PCI bridge> irq 16 at device 0.0 on pci1
pci2: <ACPI PCI bus> on pcib2
pcib3: <ACPI PCI-PCI bridge> irq 16 at device 0.0 on pci2
pci3: <ACPI PCI bus> on pcib3
pcib4: <ACPI PCI-PCI bridge> at device 0.0 on pci3
pci4: <ACPI PCI bus> on pcib4
pcib5: <ACPI PCI-PCI bridge> at device 0.2 on pci3
pci5: <ACPI PCI bus> on pcib5
pcib6: <ACPI PCI-PCI bridge> irq 18 at device 2.0 on pci2
pci6: <ACPI PCI bus> on pcib6
em0: <Intel(R) PRO/1000 Network Connection 7.3.8> port 0x2000-0x201f mem 0xd9220000-0xd923ffff,0xd9200000-0xd921ffff irq 18 at device 0.0 on pci6
em0: Using an MSI interrupt
em0: Ethernet address: 00:30:48:f2:29:9c
001.000009 netmap_attach [2244] success for em0
em1: <Intel(R) PRO/1000 Network Connection 7.3.8> port 0x2020-0x203f mem 0xd9260000-0xd927ffff,0xd9240000-0xd925ffff irq 19 at device 0.1 on pci6
em1: Using an MSI interrupt
em1: Ethernet address: 00:30:48:f2:29:9d
001.000010 netmap_attach [2244] success for em1
pcib7: <ACPI PCI-PCI bridge> at device 0.3 on pci1
pci7: <ACPI PCI bus> on pcib7
pcib8: <ACPI PCI-PCI bridge> at device 4.0 on pci0
pci8: <ACPI PCI bus> on pcib8
vgapci0: <VGA-compatible display> port 0x3000-0x307f mem 0xd8000000-0xd8ffffff,0xc0000000-0xc7ffffff,0xc8000000-0xc9ffffff irq 16 at device 0.0 on pci8
nvidia0: <GeForce GT 610> on vgapci0
vgapci0: child nvidia0 requested pci_enable_io
vgapci0: child nvidia0 requested pci_enable_io
pci8: <multimedia, HDA> at device 0.1 (no driver attached)
pcib9: <ACPI PCI-PCI bridge> at device 6.0 on pci0
pci9: <ACPI PCI bus> on pcib9
pci0: <base peripheral> at device 8.0 (no driver attached)
pcib10: <ACPI PCI-PCI bridge> irq 17 at device 28.0 on pci0
pci10: <ACPI PCI bus> on pcib10
pcib11: <PCI-PCI bridge> irq 16 at device 0.0 on pci10
pci11: <PCI bus> on pcib11
pcm0: <Envy24HT audio (Generic)> port 0x4080-0x409f,0x4000-0x407f irq 16 at device 0.0 on pci11
pcm0: system configuration
   SubVendorID: 0x1412, SubDeviceID: 0x2403
   XIN2 Clock Source: 24.576MHz(96kHz*256)
   MPU-401 UART(s) #: not implemented
   ADC #: 1 and SPDIF receiver connected
   DAC #: 4
   Multi-track converter type: AC'97(SDATA_OUT:packed)
   S/PDIF(IN/OUT): 1/1 ID# 0x00
   GPIO(mask/dir/state): 0xff/0xff/0xff
uhci0: <Intel 631XESB/632XESB/3100 USB controller USB-1> port 0x1800-0x181f irq 17 at device 29.0 on pci0
usbus0 on uhci0
uhci1: <Intel 631XESB/632XESB/3100 USB controller USB-2> port 0x1820-0x183f irq 19 at device 29.1 on pci0
usbus1 on uhci1
uhci2: <Intel 631XESB/632XESB/3100 USB controller USB-3> port 0x1840-0x185f irq 18 at device 29.2 on pci0
usbus2 on uhci2
ehci0: <Intel 63XXESB USB 2.0 controller> mem 0xd9600400-0xd96007ff irq 17 at device 29.7 on pci0
usbus3: EHCI version 1.0
usbus3 on ehci0
pcib12: <ACPI PCI-PCI bridge> at device 30.0 on pci0
pci12: <ACPI PCI bus> on pcib12
vgapci1: <VGA-compatible display> port 0x5000-0x50ff mem 0xd0000000-0xd7ffffff,0xd9300000-0xd930ffff irq 18 at device 1.0 on pci12
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel 63XXESB2 UDMA100 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x1860-0x186f at device 31.1 on pci0
ata0: <ATA channel> at channel 0 on atapci0
ahci0: <Intel ESB2 AHCI SATA controller> port 0x18a0-0x18a7,0x1874-0x1877,0x1878-0x187f,0x1870-0x1873,0x1880-0x189f mem 0xd9600800-0xd9600bff irq 19 at device 31.2 on pci0
ahci0: AHCI v1.10 with 6 3Gbps ports, Port Multiplier supported
ahcich0: <AHCI channel> at channel 0 on ahci0
ahcich1: <AHCI channel> at channel 1 on ahci0
ahcich2: <AHCI channel> at channel 2 on ahci0
ahcich3: <AHCI channel> at channel 3 on ahci0
ahcich4: <AHCI channel> at channel 4 on ahci0
ahcich5: <AHCI channel> at channel 5 on ahci0
ichsmb0: <Intel 631xESB/6321ESB (ESB2) SMBus controller> port 0x1100-0x111f irq 19 at device 31.3 on pci0
smbus0: <System Management Bus> on ichsmb0
acpi_button0: <Power Button> on acpi0
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0
fdc0: <floppy drive controller> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0
fd0: <1440-KB 3.5" drive> on fdc0 drive 0
ppc0: <Parallel port> port 0x378-0x37f,0x778-0x77f irq 7 drq 3 on acpi0
ppc0: SMC-like chipset (ECP/EPP/PS2/NIBBLE) in COMPATIBLE mode
ppc0: FIFO with 16/16/9 bytes threshold
ppbus0: <Parallel port bus> on ppc0
lpt0: <Printer> on ppbus0
lpt0: Interrupt-driven port
ppi0: <Parallel I/O> on ppbus0
ichwd0 on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
coretemp0: <CPU On-Die Thermal Sensors> on cpu0
est0: <Enhanced SpeedStep Frequency Control> on cpu0
p4tcc0: <CPU Frequency Thermal Control> on cpu0
coretemp1: <CPU On-Die Thermal Sensors> on cpu1
est1: <Enhanced SpeedStep Frequency Control> on cpu1
p4tcc1: <CPU Frequency Thermal Control> on cpu1
coretemp2: <CPU On-Die Thermal Sensors> on cpu2
est2: <Enhanced SpeedStep Frequency Control> on cpu2
p4tcc2: <CPU Frequency Thermal Control> on cpu2
coretemp3: <CPU On-Die Thermal Sensors> on cpu3
est3: <Enhanced SpeedStep Frequency Control> on cpu3
p4tcc3: <CPU Frequency Thermal Control> on cpu3
coretemp4: <CPU On-Die Thermal Sensors> on cpu4
est4: <Enhanced SpeedStep Frequency Control> on cpu4
p4tcc4: <CPU Frequency Thermal Control> on cpu4
coretemp5: <CPU On-Die Thermal Sensors> on cpu5
est5: <Enhanced SpeedStep Frequency Control> on cpu5
p4tcc5: <CPU Frequency Thermal Control> on cpu5
coretemp6: <CPU On-Die Thermal Sensors> on cpu6
est6: <Enhanced SpeedStep Frequency Control> on cpu6
p4tcc6: <CPU Frequency Thermal Control> on cpu6
coretemp7: <CPU On-Die Thermal Sensors> on cpu7
est7: <Enhanced SpeedStep Frequency Control> on cpu7
p4tcc7: <CPU Frequency Thermal Control> on cpu7
ZFS filesystem version: 5
ZFS storage pool version: features support (5000)
Timecounters tick every 1.000 msec
vboxdrv: fAsync=0 offMin=0x39c offMax=0x501
usbus0: 12Mbps Full Speed USB v1.0
usbus1: 12Mbps Full Speed USB v1.0
usbus2: 12Mbps Full Speed USB v1.0
usbus3: 480Mbps High Speed USB v2.0
ugen0.1: <Intel> at usbus0
uhub0: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0
ugen1.1: <Intel> at usbus1
uhub1: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus1
ugen2.1: <Intel> at usbus2
uhub2: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus2
ugen3.1: <Intel> at usbus3
uhub3: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus3
ata0: DMA limited to UDMA33, controller found non-ATA66 cable
ada0 at ahcich0 bus 0 scbus1 target 0 lun 0
ada0: <ST2000DL003-9VT166 CC3D> ATA-8 SATA 3.x device
ada0: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada0: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
ada0: quirks=0x1<4K>
ada0: Previously was known as ad4
ada1 at ahcich1 bus 0 scbus2 target 0 lun 0
ada1: <ST2000DL003-9VT166 CC32> ATA-8 SATA 3.x device
ada1: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada1: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
ada1: quirks=0x1<4K>
ada1: Previously was known as ad6
ada2 at ahcich2 bus 0 scbus3 target 0 lun 0
ada2: <ST2000DL003-9VT166 CC32> ATA-8 SATA 3.x device
ada2: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada2: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
ada2: quirks=0x1<4K>
ada2: Previously was known as ad8
ada3 at ahcich3 bus 0 scbus4 target 0 lun 0
ada3: <ST2000DL003-9VT166 CC32> ATA-8 SATA 3.x device
ada3: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada3: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
ada3: quirks=0x1<4K>
ada3: Previously was known as ad10
ada4 at ahcich4 bus 0 scbus5 target 0 lun 0
ada4: <ST2000DL003-9VT166 CC32> ATA-8 SATA 3.x device
ada4: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada4: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
ada4: quirks=0x1<4K>
ada4: Previously was known as ad12
ada5 at ahcich5 bus 0 scbus6 target 0 lun 0
ada5: <ST2000DL003-9VT166 CC3D> ATA-8 SATA 3.x device
ada5: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada5: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
ada5: quirks=0x1<4K>
ada5: Previously was known as ad14
cd0 at ata0 bus 0 scbus0 target 0 lun 0
cd0: <Memorex DVD+-RAM 510L v1 MWS7> Removable CD-ROM SCSI-0 device 
cd0: 33.300MB/s transfers (UDMA2, ATAPI 12bytes, PIO 65534bytes)
cd0: Attempt to query device size failed: NOT READY, Medium not present
SMP: AP CPU #7 Launched!
SMP: AP CPU #6 Launched!
SMP: AP CPU #4 Launched!
SMP: AP CPU #1 Launched!
SMP: AP CPU #2 Launched!
SMP: AP CPU #3 Launched!
SMP: AP CPU #5 Launched!
uhub0: 2 ports with 2 removable, self powered
uhub1: 2 ports with 2 removable, self powered
uhub2: 2 ports with 2 removable, self powered
Root mount waiting for: usbus3
uhub3: 6 ports with 6 removable, self powered
Root mount waiting for: usbus3
Root mount waiting for: usbus3
ugen3.2: <Peppercon AG> at usbus3
ukbd0: <Peppercon AG Multidevice, class 0/0, rev 2.00/0.01, addr 2> on usbus3
kbd2 at ukbd0
Trying to mount root from zfs:zroot/ROOT/default []...
ugen0.2: <Microsoft> at usbus0
ugen0.3: <American Power Conversion> at usbus0
ffclock reset: HPET (14318180 Hz), time = 1376183101.500000000
Setting hostuuid: 53d19f64-d663-a017-8922-0030488e9ff3.
Setting hostid: 0xf53a926e.
Entropy harvesting: interrupts ethernet point_to_point kickstart.
Starting file system checks:
Mounting local file systems:.
Writing entropy file:.
Setting hostname: borg.lerctr.org.
Starting Network: lo0 em0 em1.
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
 	options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
 	inet6 ::1 prefixlen 128
 	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
 	inet 127.0.0.1 netmask 0xff000000
 	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
 	options=4219b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_MAGIC,VLAN_HWTSO>
 	ether 00:30:48:f2:29:9c
 	inet 192.168.200.4 netmask 0xffffff00 broadcast 192.168.200.255
 	inet6 fe80::230:48ff:fef2:299c%em0 prefixlen 64 scopeid 0x1
 	nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
 	media: Ethernet autoselect
 	status: no carrier
em1: flags=8c02<BROADCAST,OACTIVE,SIMPLEX,MULTICAST> metric 0 mtu 1500
 	options=4219b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_MAGIC,VLAN_HWTSO>
 	ether 00:30:48:f2:29:9d
 	nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
 	media: Ethernet autoselect
 	status: no carrier
Starting devd.
Starting Network: em1.
em1: flags=8c02<BROADCAST,OACTIVE,SIMPLEX,MULTICAST> metric 0 mtu 1500
 	options=4219b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_MAGIC,VLAN_HWTSO>
 	ether 00:30:48:f2:29:9d
 	nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
 	media: Ethernet autoselect
 	status: no carrier
uhid0: <Peppercon AG Multidevice, class 0/0, rev 2.00/0.01, addr 2> on usbus3
ums0: <Microsoft Microsoft Trackball Optical, class 0/0, rev 1.10/1.21, addr 2> on usbus0
ums0: 5 buttons and [XYZ] coordinates ID=0
add net default: gateway 192.168.200.11
add net fe80::: gateway ::1
add net ff02::: gateway ::1
add net ::ffff:0.0.0.0: gateway ::1
add net ::0.0.0.0: gateway ::1
Mounting NFS file systems:.
ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib /usr/local/kde4/lib /usr/local/lib/compat /usr/local/lib/dbmail /usr/local/lib/event2 /usr/local/lib/pth /usr/local/lib/qt4 /usr/local/lib/virtualbox
32-bit compatibility ldconfig path: /usr/lib32 /usr/local/lib32/compat
Creating and/or trimming log files.
Starting syslogd.
Starting watchdogd.
savecore: reboot after panic: page fault
Aug 10 20:05:18 borg savecore: reboot after panic: page fault
savecore: writing core to /var/crash/vmcore.8
Writing crash summary to /var/crash/core.txt.8.
Additional ABI support: linux.
Starting rpcbind.
NFS access cache time=60
Clearing /tmp (X related).
Starting mountd.
NFSv4 is disabled
Starting nfsd.
Starting apcupsd.
Updating motd:.
Mounting late file systems:.
Starting ntpd.
Starting sshblock.
Starting smartd.
pg_ctl: another server might be running; trying to start server anyway
Updating cpucodes...
/usr/local/share/cpucontrol/m401067660F.fw: updating cpu /dev/cpuctl0 from rev 0x60c to rev 0x60f... done.
/usr/local/share/cpucontrol/m401067660F.fw: updating cpu /dev/cpuctl1 from rev 0x60c to rev 0x60f... done.
/usr/local/share/cpucontrol/m401067660F.fw: updating cpu /dev/cpuctl2 from rev 0x60c to rev 0x60f... done.
/usr/local/share/cpucontrol/m401067660F.fw: updating cpu /dev/cpuctl3 from rev 0x60c to rev 0x60f... done.
/usr/local/share/cpucontrol/m401067660F.fw: updating cpu /dev/cpuctl4 from rev 0x60c to rev 0x60f... done.
/usr/local/share/cpucontrol/m401067660F.fw: updating cpu /dev/cpuctl5 from rev 0x60c to rev 0x60f... done.
/usr/local/share/cpucontrol/m401067660F.fw: updating cpu /dev/cpuctl6 from rev 0x60c to rev 0x60f... done.
/usr/local/share/cpucontrol/m401067660F.fw: updating cpu /dev/cpuctl7 from rev 0x60c to rev 0x60f... done.
Done.
Starting exim.
Starting cupsd.
Starting boinc_client.
Starting bacula_sd.
Starting bacula_fd.
Starting bacula_dir.
Performing sanity check on sshd configuration.
Starting sshd.
Configuring syscons: blanktime.
Starting cron.
mixer: WRITE_MIXER: Device not configured
mixer: WRITE_MIXER: Device not configured
mixer: WRITE_MIXER: Device not configured
mixer: WRITE_MIXER: Device not configured
mixer: WRITE_MIXER: Device not configured
mixer: WRITE_MIXER: Device not configured
Starting inetd.
Starting background file system checks in 60 seconds.

Sat Aug 10 20:09:02 CDT 2013
Aug 10 21:05:31 borg shutdown: reboot by ler: 
Stopping inetd.
Waiting for PIDS: 1388.
Stopping cron.
Waiting for PIDS: 1366.
Stopping sshd.
Waiting for PIDS: 1354.
Stopping bacula_dir.
Waiting for PIDS: 1349.
Stopping bacula_fd.
Waiting for PIDS: 1346.
Stopping bacula_sd.
Waiting for PIDS: 1343Aug 10 21:05:35 borg bacula-dir: Shutting down Bacula service: borg-dir ...
Aug 10 21:05:35 borg bacula-fd: Shutting down Bacula service: borg-fd ...
Aug 10 21:05:35 borg bacula-sd: Shutting down Bacula service: borg-sd ...
.
Stopping boinc_client.
Waiting for PIDS: 1339.
Stopping cupsd.
Waiting for PIDS: 1333.
Stopping exim.
Waiting for PIDS: 1324.
Stopping smartd.
Waiting for PIDS: 1208.
Stopping sshblock.
Waiting for PIDS: 1204.
Stopping ntpd.
Waiting for PIDS: 1171.
Stopping apcupsd.
Waiting for PIDS: 1142, 1142.
Stopping nfsd.
Waiting for PIDS: 1132 1138, 1132.
Stopping mountd.
Waiting for PIDS: 1126.
Stopping rpcbind.
Waiting for PIDS: 1091.
Stopping watchdogd.
Waiting for PIDS: 1022.
Stopping devd.
Waiting for PIDS: 859.
Writing entropy file:.
.
Terminated
Aug 10 21:05:44 borg syslogd: exiting on signal 15
Waiting (max 60 seconds) for system process `vnlru' to stop...done
Waiting (max 60 seconds) for system process `bufdaemon' to stop...done

Waiting (max 60 seconds) for system process `syncer' to stop...Syncing disks, vnodes remaining...0 0 0 0 0 0 0 done
All buffers synced.


Fatal trap 9: general protection fault while in kernel mode
cpuid = 0; apic id = 00
instruction pointer	= 0x20:0xffffffff80568be0
stack pointer	        = 0x28:0xffffff8fcb587600
frame pointer	        = 0x28:0xffffff8fcb587610
code segment		= base 0x0, limit 0xfffff, type 0x1b
 			= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags	= interrupt enabled, resume, IOPL = 0
current process		= 1 (init)
trap number		= 9
panic: general protection fault
cpuid = 0
Uptime: 1h1m10s
Dumping 7633 out of 64479 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91%

------------------------------------------------------------------------
kernel config

options	CONFIG_AUTOGENERATED
ident	BORG-DTRACE
machine	amd64
cpu	HAMMER
makeoptions	WITH_CTF=1
makeoptions	DEBUG=-g
options	FFCLOCK
options	USB_DEBUG
options	RDRAND_RNG
options	PADLOCK_RNG
options	ATH_ENABLE_11N
options	AH_AR5416_INTERRUPT_MITIGATION
options	AH_SUPPORT_AR5416
options	IEEE80211_SUPPORT_MESH
options	IEEE80211_AMPDU_AGE
options	IEEE80211_DEBUG
options	SC_PIXEL_MODE
options	VESA
options	AHD_REG_PRETTY_PRINT
options	AHC_REG_PRETTY_PRINT
options	ATA_STATIC_ID
options	SMP
options	MALLOC_DEBUG_MAXZONES=8
options	KDB_TRACE
options	INCLUDE_CONFIG_FILE
options	DDB_CTF
options	KDTRACE_HOOKS
options	KDTRACE_FRAME
options	MAC
options	CAPABILITIES
options	CAPABILITY_MODE
options	AUDIT
options	HWPMC_HOOKS
options	KBD_INSTALL_CDEV
options	PRINTF_BUFR_SIZE=128
options	_KPOSIX_PRIORITY_SCHEDULING
options	SYSVSEM
options	SYSVMSG
options	SYSVSHM
options	STACK
options	KTRACE
options	SCSI_DELAY=5000
options	COMPAT_FREEBSD7
options	COMPAT_FREEBSD6
options	COMPAT_FREEBSD5
options	COMPAT_FREEBSD4
options	COMPAT_FREEBSD32
options	GEOM_LABEL
options	GEOM_RAID
options	GEOM_PART_GPT
options	PSEUDOFS
options	PROCFS
options	CD9660
options	MSDOSFS
options	NFS_ROOT
options	NFSLOCKD
options	NFSD
options	NFSCL
options	QUOTA
options	SCTP
options	TCP_OFFLOAD
options	INET6
options	INET
options	PREEMPTION
options	SCHED_ULE
options	NEW_PCIB
options	GEOM_PART_MBR
options	GEOM_PART_EBR_COMPAT
options	GEOM_PART_EBR
options	GEOM_PART_BSD
device	isa
device	mem
device	io
device	uart_ns8250
device	cpufreq
device	acpi
device	pci
device	fdc
device	ahci
device	ata
device	esp
device	isci
device	scbus
device	ch
device	da
device	sa
device	cd
device	pass
device	ses
device	hptnr
device	aacraid
device	atkbdc
device	atkbd
device	psm
device	kbdmux
device	vga
device	splash
device	sc
device	agp
device	uart
device	ppc
device	ppbus
device	lpt
device	ppi
device	em
device	miibus
device	cas
device	gem
device	hme
device	nfe
device	nge
device	loop
device	random
device	ether
device	vlan
device	tun
device	md
device	gif
device	faith
device	firmware
device	bpf
device	uhci
device	ohci
device	ehci
device	xhci
device	usb
device	ukbd
device	umass
device	virtio
device	virtio_pci
device	vtnet
device	virtio_blk
device	virtio_scsi
device	virtio_balloon
device	netmap

------------------------------------------------------------------------
ddb capture buffer

ddb: ddb_capture: kvm_nlist

-- 
Larry Rosenman                     http://www.lerctr.org/~ler
Phone: +1 214-642-9640                 E-Mail: ler_at_lerctr.org
US Mail: 108 Turvey Cove, Hutto, TX 78634-5688
Received on Sun Aug 11 2013 - 00:18:56 UTC

This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:40:40 UTC