Re: zfs woes

Boris Kochergin wrote:
> Duncan Young wrote:
>> Hi all,
>>
>> Again its happened (my need to restore my main zpool from backups).
>>
>> I have been having regular crashes (only recently been getting crash
>> dumps though), and in the the process
>> of upgrading to 7.1-STABLE, upgrading zpools (now 13) and zfs (now 3), I
>> now have a problem with
>> my main data zpool.
>>
>> I was having these problems before the upgrade.
>>
>> After zpool status saying that I had corrupted files (actually a zfs
>> filesystem), I tried destroying it (the zfs filesystem), in the hope
>> that this
>> may make my problems go away).  The machine hung.
>>
>> zpool scrub works, reporting 16 errors.
>> zpool status -v crashes.
>> starting multiuser hangs (probably on mounting the dud fs).
>> I can start single user, and I am currently additional backups of the
>> filesytems I can access (most off the corrupted zpool).
>> Any accessing the corrupted zfs filesystem crashes the machine.
>>
>> Now I think it's fair enough that the filesystem is corrupted (I still
>> see a need for some type of fsck), but I don't see that the
>> machine should crash because of it. Any sugestions?
>>
>> Script started on Fri Sep 11 17:22:23 2009
>> # uname -a
>> FreeBSD  7.2-STABLE FreeBSD 7.2-STABLE #8: Fri Sep 11 04:14:39 EST
>> 2009     root_at_:/usr/obj/usr/src/sys/TRIPLE0  amd64
>> # cd /usr/src/sys/TRIPLE0
>> # kgdb ./kernel.debug /var/crash/vmcore/.25
>> GNU gdb 6.1.1 [FreeBSD]
>> Copyright 2004 Free Software Foundation, Inc.
>> GDB is free software, covered by the GNU General Public License, and 
>> you are
>> welcome to change it and/or distribute copies of it under certain
>> conditions.
>> Type "show copying" to see the conditions.
>> There is absolutely no warranty for GDB.  Type "show warranty" for 
>> details.
>> This GDB was configured as "amd64-marcel-freebsd"...
>>
>> Unread portion of the kernel message buffer:
>> Copyright (c) 1992-2009 The FreeBSD Project.
>> Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
>>    The Regents of the University of California. All rights reserved.
>> FreeBSD is a registered trademark of The FreeBSD Foundation.
>> FreeBSD 7.2-STABLE #8: Fri Sep 11 04:14:39 EST 2009
>>    root_at_:/usr/obj/usr/src/sys/TRIPLE0
>> module_register: module g_label already exists!
>> Module g_label failed to register: 17
>> Timecounter "i8254" frequency 1193182 Hz quality 0
>> CPU: AMD Phenom(tm) II X4 810 Processor (2608.81-MHz K8-class CPU)
>>  Origin = "AuthenticAMD"  Id = 0x100f42  Stepping = 2
>>
>> Features=0x178bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2,HTT> 
>>
>>  Features2=0x802009<SSE3,MON,CX16,POPCNT>
>>  AMD
>> Features=0xee500800<SYSCALL,NX,MMX+,FFXSR,Page1GB,RDTSCP,LM,3DNow!+,3DNow!> 
>>
>>  AMD
>> Features2=0x37ff<LAHF,CMP,SVM,ExtAPIC,CR8,ABM,SSE4A,MAS,Prefetch,OSVW,IBS,SKINIT,WDT> 
>>
>>  TSC: P-state invariant
>>  Cores per package: 4
>> usable memory = 8441266176 (8050 MB)
>> avail memory  = 8131702784 (7754 MB)
>> ACPI APIC Table: <072709 APIC1045>
>> FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
>> cpu0 (BSP): APIC ID:  0
>> cpu1 (AP): APIC ID:  1
>> cpu2 (AP): APIC ID:  2
>> cpu3 (AP): APIC ID:  3
>> This module (opensolaris) contains code covered by the
>> Common Development and Distribution License (CDDL)
>> see http://opensolaris.org/os/licensing/opensolaris_license/
>> ACPI Warning (tbfadt-0505): Optional field "Pm2ControlBlock" has zero
>> address or length:        0       0/1 [20070320]
>> ioapic0 <Version 2.1> irqs 0-23 on motherboard
>> kbd1 at kbdmux0
>> acpi0: <072709 XSDT1045> on motherboard
>> acpi0: [ITHREAD]
>> acpi0: Power Button (fixed)
>> acpi0: reservation of fee00000, 1000 (3) failed
>> acpi0: reservation of ffb80000, 80000 (3) failed
>> acpi0: reservation of fec10000, 20 (3) failed
>> acpi0: reservation of 0, a0000 (3) failed
>> acpi0: reservation of 100000, dff00000 (3) failed
>> ACPI HPET table warning: Sequence is non-zero (2)
>> Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
>> acpi_timer0: <32-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0
>> acpi_hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on
>> acpi0
>> Timecounter "HPET" frequency 14318180 Hz quality 900
>> pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
>> pci0: <ACPI PCI bus> on pcib0
>> pcib1: <ACPI PCI-PCI bridge> at device 1.0 on pci0
>> pci1: <ACPI PCI bus> on pcib1
>> vgapci0: <VGA-compatible display> port 0xc000-0xc0ff mem
>> 0xf0000000-0xf7ffffff,0xfbce0000-0xfbceffff,0xfbb00000-0xfbbfffff irq 18
>> at device 5.0 on pci1
>> hdac0: <ATI RS780 High Definition Audio Controller> mem
>> 0xfbcfc000-0xfbcfffff irq 19 at device 5.1 on pci1
>> hdac0: HDA Driver Revision: 20090624_0136
>> hdac0: [ITHREAD]
>> pcib2: <ACPI PCI-PCI bridge> irq 16 at device 4.0 on pci0
>> pci2: <ACPI PCI bus> on pcib2
>> hptrr0: <sx508x> port 0xd800-0xd8ff mem 0xfbe00000-0xfbefffff irq 16 at
>> device 0.0 on pci2
>> hptrr: adapter at PCI 2:0:0, IRQ 16
>> pcib3: <ACPI PCI-PCI bridge> irq 17 at device 5.0 on pci0
>> pci3: <ACPI PCI bus> on pcib3
>> em0: <Intel(R) PRO/1000 Network Connection 6.9.6> port 0xec00-0xec1f mem
>> 0xfbfe0000-0xfbffffff,0xfbf00000-0xfbf7ffff,0xfbfdc000-0xfbfdffff irq 17
>> at device 0.0 on pci3
>> em0: Using MSIX interrupts
>> em0: [ITHREAD]
>> em0: [ITHREAD]
>> em0: [ITHREAD]
>> em0: Ethernet address: 00:1b:21:3d:eb:e7
>> atapci0: <ATI AHCI controller> port
>> 0xb000-0xb007,0xa000-0xa003,0x9000-0x9007,0x8000-0x8003,0x7000-0x700f
>> mem 0xfbaffc00-0xfbafffff irq 22 at device 17.0 on pci0
>> atapci0: [ITHREAD]
>> atapci0: AHCI Version 01.10 controller with 6 ports detected
>> ata2: <ATA channel 0> on atapci0
>> ata2: [ITHREAD]
>> ata3: <ATA channel 1> on atapci0
>> ata3: [ITHREAD]
>> ata4: <ATA channel 2> on atapci0
>> ata4: [ITHREAD]
>> ata5: <ATA channel 3> on atapci0
>> ata5: [ITHREAD]
>> ata6: <ATA channel 4> on atapci0
>> ata6: [ITHREAD]
>> ata7: <ATA channel 5> on atapci0
>> ata7: [ITHREAD]
>> ohci0: <OHCI (generic) USB controller> mem 0xfbafd000-0xfbafdfff irq 16
>> at device 18.0 on pci0
>> ohci0: [GIANT-LOCKED]
>> ohci0: [ITHREAD]
>> usb0: OHCI version 1.0, legacy support
>> usb0: <OHCI (generic) USB controller> on ohci0
>> usb0: USB revision 1.0
>> uhub0: <ATI OHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb0
>> uhub0: 3 ports with 3 removable, self powered
>> ohci1: <OHCI (generic) USB controller> mem 0xfbafe000-0xfbafefff irq 16
>> at device 18.1 on pci0
>> ohci1: [GIANT-LOCKED]
>> ohci1: [ITHREAD]
>> usb1: OHCI version 1.0, legacy support
>> usb1: <OHCI (generic) USB controller> on ohci1
>> usb1: USB revision 1.0
>> uhub1: <ATI OHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb1
>> uhub1: 3 ports with 3 removable, self powered
>> ehci0: <EHCI (generic) USB 2.0 controller> mem 0xfbaff800-0xfbaff8ff irq
>> 17 at device 18.2 on pci0
>> ehci0: [GIANT-LOCKED]
>> ehci0: [ITHREAD]
>> usb2: EHCI version 1.0
>> usb2: companion controllers, 3 ports each: usb0 usb1
>> usb2: <EHCI (generic) USB 2.0 controller> on ehci0
>> usb2: USB revision 2.0
>> uhub2: <ATI EHCI root hub, class 9/0, rev 2.00/1.00, addr 1> on usb2
>> uhub2: 6 ports with 6 removable, self powered
>> ohci2: <OHCI (generic) USB controller> mem 0xfbafb000-0xfbafbfff irq 18
>> at device 19.0 on pci0
>> ohci2: [GIANT-LOCKED]
>> ohci2: [ITHREAD]
>> usb3: OHCI version 1.0, legacy support
>> usb3: <OHCI (generic) USB controller> on ohci2
>> usb3: USB revision 1.0
>> uhub3: <ATI OHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb3
>> uhub3: 3 ports with 3 removable, self powered
>> ohci3: <OHCI (generic) USB controller> mem 0xfbafc000-0xfbafcfff irq 18
>> at device 19.1 on pci0
>> ohci3: [GIANT-LOCKED]
>> ohci3: [ITHREAD]
>> usb4: OHCI version 1.0, legacy support
>> usb4: <OHCI (generic) USB controller> on ohci3
>> usb4: USB revision 1.0
>> uhub4: <ATI OHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb4
>> uhub4: 3 ports with 3 removable, self powered
>> ehci1: <EHCI (generic) USB 2.0 controller> mem 0xfbaff400-0xfbaff4ff irq
>> 19 at device 19.2 on pci0
>> ehci1: [GIANT-LOCKED]
>> ehci1: [ITHREAD]
>> usb5: EHCI version 1.0
>> usb5: companion controllers, 3 ports each: usb3 usb4
>> usb5: <EHCI (generic) USB 2.0 controller> on ehci1
>> usb5: USB revision 2.0
>> uhub5: <ATI EHCI root hub, class 9/0, rev 2.00/1.00, addr 1> on usb5
>> uhub5: 6 ports with 6 removable, self powered
>> pci0: <serial bus, SMBus> at device 20.0 (no driver attached)
>> atapci1: <ATI IXP700 UDMA133 controller> port
>> 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xff00-0xff0f at device 20.1 on pci0
>> ata0: <ATA channel 0> on atapci1
>> ata0: [ITHREAD]
>> hdac1: <ATI SB600 High Definition Audio Controller> mem
>> 0xfbaf4000-0xfbaf7fff irq 16 at device 20.2 on pci0
>> hdac1: HDA Driver Revision: 20090624_0136
>> hdac1: [ITHREAD]
>> isab0: <PCI-ISA bridge> at device 20.3 on pci0
>> isa0: <ISA bus> on isab0
>> pcib4: <ACPI PCI-PCI bridge> at device 20.4 on pci0
>> pci4: <ACPI PCI bus> on pcib4
>> ohci4: <OHCI (generic) USB controller> mem 0xfbafa000-0xfbafafff irq 18
>> at device 20.5 on pci0
>> ohci4: [GIANT-LOCKED]
>> ohci4: [ITHREAD]
>> usb6: OHCI version 1.0, legacy support
>> usb6: <OHCI (generic) USB controller> on ohci4
>> usb6: USB revision 1.0
>> uhub6: <ATI OHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb6
>> uhub6: 2 ports with 2 removable, self powered
>> acpi_button0: <Power Button> on acpi0
>> atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
>> atkbd0: <AT Keyboard> irq 1 on atkbdc0
>> kbd0 at atkbd0
>> atkbd0: [GIANT-LOCKED]
>> atkbd0: [ITHREAD]
>> psm0: <PS/2 Mouse> irq 12 on atkbdc0
>> psm0: [GIANT-LOCKED]
>> psm0: [ITHREAD]
>> psm0: model MouseMan+, device ID 0
>> acpi_aiboost0: <ASUStek AIBOOSTER> on acpi0
>> sio0: configured irq 4 not in bitmap of probed irqs 0
>> sio0: port may not be enabled
>> sio0: configured irq 4 not in bitmap of probed irqs 0
>> sio0: port may not be enabled
>> sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 flags 0x10 on
>> acpi0
>> sio0: type 16550A
>> sio0: [FILTER]
>> fdc0: <floppy drive controller (FDE)> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2
>> on acpi0
>> fdc0: [FILTER]
>> cpu0: <ACPI CPU> on acpi0
>> acpi_throttle0: <ACPI CPU Throttling> on cpu0
>> cpu1: <ACPI CPU> on acpi0
>> cpu2: <ACPI CPU> on acpi0
>> cpu3: <ACPI CPU> on acpi0
>> cryptosoft0: <software crypto> on motherboard
>> orm0: <ISA Option ROM> at iomem 0xd7800-0xd87ff on isa0
>> sc0: <System console> at flags 0x100 on isa0
>> sc0: VGA <16 virtual consoles, flags=0x300>
>> vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on 
>> isa0
>> ppc0: cannot reserve I/O port range
>> sio1: configured irq 3 not in bitmap of probed irqs 0
>> sio1: port may not be enabled
>> ulpt0: <Samsung Electronics Co., Ltd. Samsung Laser Printer ML-1200
>> Series, class 0/0, rev 1.10/1.00, addr 2> on uhub0
>> ulpt0: using bi-directional mode
>> WARNING: ZFS is considered to be an experimental feature in FreeBSD.
>> Timecounters tick every 1.000 msec
>> vboxdrv: fAsync=0 offMin=0x12c offMax=0x219
>> hptrr: start channel [0,0]
>> hptrr: start channel [0,1]
>> hptrr: start channel [0,2]
>> hptrr: start channel [0,3]
>> hptrr: [0 0] Start channel soft reset.
>> hptrr: [0 2] Start channel soft reset.
>> hptrr: [0 3] Start channel soft reset.
>> hptrr: channel [0,0] started successfully
>> hptrr: channel [0,2] started successfully
>> hptrr: channel [0,3] started successfully
>> hptrr: [0 1] Failed to perform channel hard reset.
>> hptrr0: [GIANT-LOCKED]
>> hptrr0: [ITHREAD]
>> The GEOM class LABEL is already loaded.
>> ZFS filesystem version 13
>> ZFS storage pool version 13
>> da0 at hptrr0 bus 0 target 0 lun 0
>> da0: <HPT DISK 0_0 4.00> Fixed Direct Access SCSI-0 device
>> da1 at hptrr0 bus 0 target 1 lun 0
>> da1: <HPT DISK 0_1 4.00> Fixed Direct Access SCSI-0 device
>> da2 at hptrr0 bus 0 target 2 lun 0
>> da2: <HPT DISK 0_2 4.00> Fixed Direct Access SCSI-0 device
>> acd0: DVDR <HL-DT-ST DVDRAM GSA-4163B/A104> at ata0-master PIO4
>> ad4: 476940MB <WDC WD5000AAKS-00C8A0 12.01C02> at ata2-master SATA300
>> acd0: FAILURE - INQUIRY ILLEGAL REQUEST asc=0x24 ascq=0x00 sks=0x40 0x00
>> 0x01
>> (probe0:ata0:0:0:0): TEST UNIT READY. CDB: 0 0 0 0 0 0
>> (probe0:ata0:0:0:0): CAM Status: SCSI Status Error
>> (probe0:ata0:0:0:0): SCSI Status: Check Condition
>> (probe0:ata0:0:0:0): NOT READY csi:0,0,bb,0 asc:3a,0
>> (probe0:ata0:0:0:0): Medium not present
>> (probe0:ata0:0:0:0): Unretryable error
>> cd0 at ata0 bus 0 target 0 lun 0
>> cd0: <HL-DT-ST DVDRAM GSA-4163B A104> Removable CD-ROM SCSI-0 device
>> cd0: 16.000MB/s transfers
>> cd0: Attempt to query device size failed: NOT READY, Medium not present
>> ad6: 953869MB <WDC WD10EADS-00L5B1 01.01A01> at ata3-master SATA300
>> ad10: 953869MB <WDC WD10EACS-00D6B0 01.01A01> at ata5-master SATA300
>> ad12: 476940MB <WDC WD5000AACS-00ZUB0 01.01B01> at ata6-master SATA300
>> ad14: 953869MB <Seagate ST31000528AS CC44> at ata7-master SATA300
>> hdac0: HDA Codec #0: ATI RS690/780 HDMI
>> pcm0: <HDA ATI RS690/780 HDMI PCM #0 Digital> at cad 0 nid 1 on hdac0
>> hdac1: HDA Codec #0: VIA VT1708S_0
>> pcm1: <HDA VIA VT1708S_0 PCM #0 Analog> at cad 0 nid 1 on hdac1
>> pcm2: <HDA VIA VT1708S_0 PCM #1 Digital> at cad 0 nid 1 on hdac1
>> pcm3: <HDA VIA VT1708S_0 PCM #2 Digital> at cad 0 nid 1 on hdac1
>> SMP: AP CPU #2 Launched!
>> SMP: AP CPU #3 Launched!
>> SMP: AP CPU #1 Launched!
>> Trying to mount root from zfs:rootzfs
>> <118>Loading configuration files.
>> <118>kernel dumps on /dev/ad4s1b
>> <118>Entropy harvesting:
>> <118> interrupts
>> <118> ethernet
>> <118> point_to_point
>> <118> kickstart
>> <118>.
>> GEOM_ELI: Device label/swapB.eli created.
>> GEOM_ELI: Encryption: AES-CBC 256
>> GEOM_ELI:     Crypto: software
>> <118>swapon: adding /dev/label/swapB.eli as swap device
>> <118>/etc/rc: WARNING: Use of the early.sh script is deprecated
>> <118>/etc/rc: WARNING: Please use a new-style rc.d script instead
>> <118>/etc/rc: WARNING: See rc(8) for more information
>> <118>gmirror:
>> <118>No such device: swap_AB.
>> <118>
>> <118>Starting file system checks:
>> <118>/dev/label/bstrapA: FILE SYSTEM CLEAN; SKIPPING CHECKS
>> <118>/dev/label/bstrapA: clean, 303581 free (261 frags, 37915 blocks,
>> 0.1% fragmentation)
>> <118>/dev/label/bstrapB: FILE SYSTEM CLEAN; SKIPPING CHECKS
>> <118>/dev/label/bstrapB: clean, 303502 free (318 frags, 37898 blocks,
>> 0.1% fragmentation)
>> <118>Setting hostuuid: e039c626-5f54-da11-8e5a-e6b6a8a23cf0.
>> <118>Setting hostid: 0xd449202f.
>> <118>Mounting local file systems:
>> <118>.
>>
>>
>> Fatal trap 12: page fault while in kernel mode
>> cpuid = 1; apic id = 01
>> fault virtual address    = 0x50
>> fault code        = supervisor read data, page not present
>> instruction pointer    = 0x8:0xffffffff80ad7a5e
>> stack pointer            = 0x10:0xffffff80a5b2f620
>> frame pointer            = 0x10:0xffffff80a5b2f6a0
>> code segment        = base 0x0, limit 0xfffff, type 0x1b
>>            = DPL 0, pres 1, long 1, def32 0, gran 1
>> processor eflags    = interrupt enabled, resume, IOPL = 0
>> current process        = 314 (txg_thread_enter)
>> trap number        = 12
>> panic: page fault
>> cpuid = 1
>> Uptime: 2m11s
>> Physical memory: 8050 MB
>> Dumping 1553 MB: 1538 1522 1506 1490 1474 1458 1442 1426 1410 1394 1378
>> 1362 1346 1330 1314 1298 1282 1266 1250 1234 1218 1202 1186 1170 1154
>> 1138 1122 1106 1090 1074 1058 1042 1026 1010 994 978 962 946 930 914 898
>> 882 866 850 834 818 802 786 770 754 738 722 706 690 674 658 642 626 610
>> 594 578 562 546 530 514 498 482 466 450 434 418 402 386 370 354 338 322
>> 306 290 274 258 242 226 210 194 178 162 146 130 114 98 82 66 50 34 18 2
>>
>> Reading symbols from /boot/kernel/zfs.ko...Reading symbols from
>> /bootdirA/boot/kernel/zfs.ko.symbols...done.
>> done.
>> Loaded symbols for /boot/kernel/zfs.ko
>> Reading symbols from /boot/kernel/opensolaris.ko...Reading symbols from
>> /bootdirA/boot/kernel/opensolaris.ko.symbols...done.
>> done.
>> Loaded symbols for /boot/kernel/opensolaris.ko
>> Reading symbols from /boot/kernel/geom_eli.ko...Reading symbols from
>> /bootdirA/boot/kernel/geom_eli.ko.symbols...done.
>> done.
>> Loaded symbols for /boot/kernel/geom_eli.ko
>> Reading symbols from /boot/kernel/crypto.ko...Reading symbols from
>> /bootdirA/boot/kernel/crypto.ko.symbols...done.
>> done.
>> Loaded symbols for /boot/kernel/crypto.ko
>> Reading symbols from /boot/kernel/zlib.ko...Reading symbols from
>> /bootdirA/boot/kernel/zlib.ko.symbols...done.
>> done.
>> Loaded symbols for /boot/kernel/zlib.ko
>> Reading symbols from /boot/kernel/geom_label.ko...Reading symbols from
>> /bootdirA/boot/kernel/geom_label.ko.symbols...done.
>> done.
>> Loaded symbols for /boot/kernel/geom_label.ko
>> Reading symbols from /boot/kernel/geom_mirror.ko...Reading symbols from
>> /bootdirA/boot/kernel/geom_mirror.ko.symbols...done.
>> done.
>> Loaded symbols for /boot/kernel/geom_mirror.ko
>> Reading symbols from /boot/kernel/snd_hda.ko...Reading symbols from
>> /bootdirA/boot/kernel/snd_hda.ko.symbols...done.
>> done.
>> Loaded symbols for /boot/kernel/snd_hda.ko
>> Reading symbols from /boot/kernel/sound.ko...Reading symbols from
>> /bootdirA/boot/kernel/sound.ko.symbols...done.
>> done.
>> Loaded symbols for /boot/kernel/sound.ko
>> Reading symbols from /boot/kernel/cd9660_iconv.ko...Reading symbols from
>> /bootdirA/boot/kernel/cd9660_iconv.ko.symbols...done.
>> done.
>> Loaded symbols for /boot/kernel/cd9660_iconv.ko
>> Reading symbols from /boot/kernel/libiconv.ko...Reading symbols from
>> /bootdirA/boot/kernel/libiconv.ko.symbols...done.
>> done.
>> Loaded symbols for /boot/kernel/libiconv.ko
>> Reading symbols from /boot/kernel/aio.ko...Reading symbols from
>> /bootdirA/boot/kernel/aio.ko.symbols...done.
>> done.
>> Loaded symbols for /boot/kernel/aio.ko
>> Reading symbols from /boot/modules/vboxdrv.ko...done.
>> Loaded symbols for /boot/modules/vboxdrv.ko
>> Reading symbols from /boot/kernel/sem.ko...Reading symbols from
>> /bootdirA/boot/kernel/sem.ko.symbols...done.
>> done.
>> Loaded symbols for /boot/kernel/sem.ko
>> #0  doadump () at pcpu.h:195
>> 195        __asm __volatile("movq %%gs:0,%0" : "=r" (td));
>> (kgdb) backtrace
>> #0  doadump () at pcpu.h:195
>> #1  0x0000000000000004 in ?? ()
>> #2  0xffffffff803aab90 in boot (howto=260)
>>    at /usr/src/sys/kern/kern_shutdown.c:418
>> #3  0xffffffff803aafb2 in panic (fmt=0x104 <Address 0x104 out of 
>> bounds>)
>>    at /usr/src/sys/kern/kern_shutdown.c:574
>> #4  0xffffffff80651d03 in trap_fatal (frame=0xffffff0009572390,
>> eva=Variable "eva" is not available.
>> )
>>    at /usr/src/sys/amd64/amd64/trap.c:770
>> #5  0xffffffff806520d5 in trap_pfault (frame=0xffffff80a5b2f570, 
>> usermode=0)
>>    at /usr/src/sys/amd64/amd64/trap.c:686
>> #6  0xffffffff80652a63 in trap (frame=0xffffff80a5b2f570)
>>    at /usr/src/sys/amd64/amd64/trap.c:457
>> #7  0xffffffff8063cb4e in calltrap ()
>>    at /usr/src/sys/amd64/amd64/exception.S:218
>> #8  0xffffffff80ad7a5e in find_block (th=0xffffff8006930000,
>>    zseg=0xffffff000dafc580, dnp=0xffffff8006d35600, depth=Variable
>> "depth" is not available.
>> )
>>    at
>> /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_traverse.c:400 
>>
>> #9  0xffffffff80ad81a3 in traverse_more (th=0xffffff8006930000)
>>    at
>> /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_traverse.c:672 
>>
>> #10 0xffffffff80ad8368 in traverse_dsl_dataset (ds=0x23, txg_start=1,
>> advance=Variable "advance" is not available.
>>
>> )
>>    at
>> /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/d---Type 
>>
>> <return> to continue, or q <return> to quit---
>> mu_traverse.c:731
>> #11 0xffffffff80ae41c9 in dsl_dataset_destroy_sync () from
>> /boot/kernel/zfs.ko
>> #12 0xffffffff80ae6d2a in dsl_sync_task_group_sync
>> (dstg=0xffffff000d256600,
>>    tx=0xffffff000d1fca80)
>>    at
>> /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_synctask.c:186 
>>
>> #13 0xffffffff80ae6873 in dsl_pool_sync (dp=0xffffff000d521000, 
>> txg=3430336)
>>    at
>> /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_pool.c:316 
>>
>> #14 0xffffffff80af59c3 in spa_sync (spa=0xffffff0006d08000, txg=3430336)
>>    at
>> /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/spa.c:3988 
>>
>> #15 0xffffffff80afd772 in txg_sync_thread (arg=Variable "arg" is not
>> available.
>> )
>>    at
>> /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/txg.c:352 
>>
>> #16 0xffffffff80385452 in fork_exit (
>>    callout=0xffffffff80afd490 <txg_sync_thread>, arg=0xffffff000d521000,
>>    frame=0xffffff80a5b2fc80) at /usr/src/sys/kern/kern_fork.c:811
>> #17 0xffffffff8063cf2e in fork_trampoline ()
>>    at /usr/src/sys/amd64/amd64/exception.S:554
>> #18 0x0000000000000000 in ?? ()
>> #19 0x0000000000000000 in ?? ()
>> #20 0x0000000000000001 in ?? ()
>> #21 0x0000000000000000 in ?? ()
>> #22 0x0000000000000000 in ?? ()
>> #23 0x0000000000000000 in ?? ()
>> #24 0x0000000000000000 in ?? ()
>> #25 0x0000000000000000 in ?? ()
>> #26 0x0000000000000000 in ?? ()
>> #27 0x0000000000000000 in ?? ()
>> #28 0x0000000000000000 in ?? ()
>> #29 0x0000000000000000 in ?? ()
>> #30 0x0000000000000000 in ?? ()
>> #31 0x0000000000000000 in ?? ()
>> #32 0x0000000000000000 in ?? ()
>> #33 0x0000000000000000 in ?? ()
>> #34 0x0000000000000000 in ?? ()
>> #35 0x0000000000000000 in ?? ()
>> #36 0x0000000000000000 in ?? ()
>> #37 0x0000000000000000 in ?? ()
>> #38 0x0000000000000000 in ?? ()
>> #39 0x0000000000000000 in ?? ()
>> #40 0x0000000000000000 in ?? ()
>> #41 0x0000000000000000 in ?? ()
>> #42 0x0000000000dde000 in ?? ()
>> #43 0xffffffff80950d40 in tdq_cpu ()
>> #44 0xffffffff8095c940 in tdq_groups ()
>> #45 0xffffffff8095c8c0 in tdq_cpu ()
>> #46 0xffffff0009572390 in ?? ()
>> #47 0x0000000000000000 in ?? ()
>> #48 0xffffff80a5b2f1e8 in ?? ()
>> #49 0xffffff0009572390 in ?? ()
>> #50 0xffffffff803ce154 in sched_switch (td=0xffffffff80afd490, 
>> newtd=0x0,
>>    flags=0) at /usr/src/sys/kern/sched_ule.c:1938
>> #51 0x0000000000000000 in ?? ()
>> #52 0x0000000000000000 in ?? ()
>> #53 0x0000000000000000 in ?? ()
>> #54 0x0000000000000000 in ?? ()
>> #55 0x0000000000000000 in ?? ()
>> #56 0x0000000000000000 in ?? ()
>> #57 0x0000000000000000 in ?? ()
>> #58 0x0000000000000000 in ?? ()
>> #59 0x0000000000000000 in ?? ()
>> #60 0x0000000000000000 in ?? ()
>> #61 0x0000000000000000 in ?? ()
>> #62 0x0000000000000000 in ?? ()
>> #63 0x0000000000000000 in ?? ()
>> #64 0x0000000000000000 in ?? ()
>> #65 0x0000000000000000 in ?? ()
>> #66 0x0000000000000000 in ?? ()
>> #67 0x0000000000000000 in ?? ()
>> #68 0x0000000000000000 in ?? ()
>> #69 0x0000000000000000 in ?? ()
>> #70 0x0000000000000000 in ?? ()
>> #71 0x0000000000000000 in ?? ()
>> #72 0x0000000000000000 in ?? ()
>> #73 0x0000000000000000 in ?? ()
>> #74 0x0000000000000000 in ?? ()
>> #75 0x0000000000000000 in ?? ()
>> #76 0x0000000000000000 in ?? ()
>> #77 0x0000000000000000 in ?? ()
>> #78 0x0000000000000000 in ?? ()
>> #79 0x0000000000000000 in ?? ()
>> #80 0x0000000000000000 in ?? ()
>> #81 0x0000000000000000 in ?? ()
>> #82 0x0000000000000000 in ?? ()
>> #83 0x0000000000000000 in ?? ()
>> #84 0x0000000000000000 in ?? ()
>> #85 0x0000000000000000 in ?? ()
>> #86 0x0000000000000000 in ?? ()
>> #87 0x0000000000000000 in ?? ()
>> #88 0x0000000000000000 in ?? ()
>> #89 0x0000000000000000 in ?? ()
>> #90 0x0000000000000000 in ?? ()
>> #91 0x0000000000000000 in ?? ()
>> #92 0x0000000000000000 in ?? ()
>> #93 0x0000000000000000 in ?? ()
>> #94 0x0000000000000000 in ?? ()
>> #95 0x0000000000000000 in ?? ()
>> #96 0x0000000000000000 in ?? ()
>> #97 0x0000000000000000 in ?? ()
>> #98 0x0000000000000000 in ?? ()
>> #99 0x0000000000000000 in ?? ()
>> #100 0x0000000000000000 in ?? ()
>> #101 0x0000000000000000 in ?? ()
>> #102 0x0000000000000000 in ?? ()
>> #103 0x0000000000000000 in ?? ()
>> #104 0x0000000000000000 in ?? ()
>> #105 0x0000000000000000 in ?? ()
>> #106 0x0000000000000000 in ?? ()
>> #107 0x0000000000000000 in ?? ()
>> #108 0x0000000000000000 in ?? ()
>> #109 0x0000000000000000 in ?? ()
>> #110 0x0000000000000000 in ?? ()
>> #111 0x0000000000000000 in ?? ()
>> #112 0x0000000000000000 in ?? ()
>> #113 0x0000000000000000 in ?? ()
>> #114 0x0000000000000000 in ?? ()
>> #115 0x0000000000000000 in ?? ()
>> #116 0x0000000000000000 in ?? ()
>> #117 0x0000000000000000 in ?? ()
>> #118 0x0000000000000000 in ?? ()
>> Cannot access memory at address 0xffffff80a5b30000
>> (kgdb) q
>> # ^D
>> Script done on Fri Sep 11 17:26:02 2009
>>
>> _______________________________________________
>> freebsd-current_at_freebsd.org mailing list
>> http://lists.freebsd.org/mailman/listinfo/freebsd-current
>> To unsubscribe, send any mail to 
>> "freebsd-current-unsubscribe_at_freebsd.org"
> Is the hardware in the machine (CPU, motherboard, and memory) 
> confirmed to be solid? I ran into a similar situation last month 
> (http://lists.freebsd.org/pipermail/freebsd-fs/2009-August/006606.html), 
> the root cause of which turned out to be bad memory. ZFS tends to make 
> the problem show up more frequently since it uses a lot of memory.
>
> -Boris
I believe the memory to be fine.  I ran memtest across it and found no 
errors.  Its also ECC memory (which I have switched on), so much less 
likely to have errors showing up.

Duncan