Re: zfs woes

Duncan Young wrote:
> Hi all,
>
> Again its happened (my need to restore my main zpool from backups).
>
> I have been having regular crashes (only recently been getting crash
> dumps though), and in the the process
> of upgrading to 7.1-STABLE, upgrading zpools (now 13) and zfs (now 3), I
> now have a problem with
> my main data zpool.
>
> I was having these problems before the upgrade.
>
> After zpool status saying that I had corrupted files (actually a zfs
> filesystem), I tried destroying it (the zfs filesystem), in the hope
> that this
> may make my problems go away).  The machine hung.
>
> zpool scrub works, reporting 16 errors.
> zpool status -v crashes.
> starting multiuser hangs (probably on mounting the dud fs).
> I can start single user, and I am currently additional backups of the
> filesytems I can access (most off the corrupted zpool).
> Any accessing the corrupted zfs filesystem crashes the machine.
>
> Now I think it's fair enough that the filesystem is corrupted (I still
> see a need for some type of fsck), but I don't see that the
> machine should crash because of it. Any sugestions?
>
> Script started on Fri Sep 11 17:22:23 2009
> # uname -a
> FreeBSD  7.2-STABLE FreeBSD 7.2-STABLE #8: Fri Sep 11 04:14:39 EST
> 2009     root_at_:/usr/obj/usr/src/sys/TRIPLE0  amd64
> # cd /usr/src/sys/TRIPLE0
> # kgdb ./kernel.debug /var/crash/vmcore/.25
> GNU gdb 6.1.1 [FreeBSD]
> Copyright 2004 Free Software Foundation, Inc.
> GDB is free software, covered by the GNU General Public License, and 
> you are
> welcome to change it and/or distribute copies of it under certain
> conditions.
> Type "show copying" to see the conditions.
> There is absolutely no warranty for GDB.  Type "show warranty" for 
> details.
> This GDB was configured as "amd64-marcel-freebsd"...
>
> Unread portion of the kernel message buffer:
> Copyright (c) 1992-2009 The FreeBSD Project.
> Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
>    The Regents of the University of California. All rights reserved.
> FreeBSD is a registered trademark of The FreeBSD Foundation.
> FreeBSD 7.2-STABLE #8: Fri Sep 11 04:14:39 EST 2009
>    root_at_:/usr/obj/usr/src/sys/TRIPLE0
> module_register: module g_label already exists!
> Module g_label failed to register: 17
> Timecounter "i8254" frequency 1193182 Hz quality 0
> CPU: AMD Phenom(tm) II X4 810 Processor (2608.81-MHz K8-class CPU)
>  Origin = "AuthenticAMD"  Id = 0x100f42  Stepping = 2
>
> Features=0x178bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2,HTT> 
>
>  Features2=0x802009<SSE3,MON,CX16,POPCNT>
>  AMD
> Features=0xee500800<SYSCALL,NX,MMX+,FFXSR,Page1GB,RDTSCP,LM,3DNow!+,3DNow!> 
>
>  AMD
> Features2=0x37ff<LAHF,CMP,SVM,ExtAPIC,CR8,ABM,SSE4A,MAS,Prefetch,OSVW,IBS,SKINIT,WDT> 
>
>  TSC: P-state invariant
>  Cores per package: 4
> usable memory = 8441266176 (8050 MB)
> avail memory  = 8131702784 (7754 MB)
> ACPI APIC Table: <072709 APIC1045>
> FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
> cpu0 (BSP): APIC ID:  0
> cpu1 (AP): APIC ID:  1
> cpu2 (AP): APIC ID:  2
> cpu3 (AP): APIC ID:  3
> This module (opensolaris) contains code covered by the
> Common Development and Distribution License (CDDL)
> see http://opensolaris.org/os/licensing/opensolaris_license/
> ACPI Warning (tbfadt-0505): Optional field "Pm2ControlBlock" has zero
> address or length:        0       0/1 [20070320]
> ioapic0 <Version 2.1> irqs 0-23 on motherboard
> kbd1 at kbdmux0
> acpi0: <072709 XSDT1045> on motherboard
> acpi0: [ITHREAD]
> acpi0: Power Button (fixed)
> acpi0: reservation of fee00000, 1000 (3) failed
> acpi0: reservation of ffb80000, 80000 (3) failed
> acpi0: reservation of fec10000, 20 (3) failed
> acpi0: reservation of 0, a0000 (3) failed
> acpi0: reservation of 100000, dff00000 (3) failed
> ACPI HPET table warning: Sequence is non-zero (2)
> Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
> acpi_timer0: <32-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0
> acpi_hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on
> acpi0
> Timecounter "HPET" frequency 14318180 Hz quality 900
> pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
> pci0: <ACPI PCI bus> on pcib0
> pcib1: <ACPI PCI-PCI bridge> at device 1.0 on pci0
> pci1: <ACPI PCI bus> on pcib1
> vgapci0: <VGA-compatible display> port 0xc000-0xc0ff mem
> 0xf0000000-0xf7ffffff,0xfbce0000-0xfbceffff,0xfbb00000-0xfbbfffff irq 18
> at device 5.0 on pci1
> hdac0: <ATI RS780 High Definition Audio Controller> mem
> 0xfbcfc000-0xfbcfffff irq 19 at device 5.1 on pci1
> hdac0: HDA Driver Revision: 20090624_0136
> hdac0: [ITHREAD]
> pcib2: <ACPI PCI-PCI bridge> irq 16 at device 4.0 on pci0
> pci2: <ACPI PCI bus> on pcib2
> hptrr0: <sx508x> port 0xd800-0xd8ff mem 0xfbe00000-0xfbefffff irq 16 at
> device 0.0 on pci2
> hptrr: adapter at PCI 2:0:0, IRQ 16
> pcib3: <ACPI PCI-PCI bridge> irq 17 at device 5.0 on pci0
> pci3: <ACPI PCI bus> on pcib3
> em0: <Intel(R) PRO/1000 Network Connection 6.9.6> port 0xec00-0xec1f mem
> 0xfbfe0000-0xfbffffff,0xfbf00000-0xfbf7ffff,0xfbfdc000-0xfbfdffff irq 17
> at device 0.0 on pci3
> em0: Using MSIX interrupts
> em0: [ITHREAD]
> em0: [ITHREAD]
> em0: [ITHREAD]
> em0: Ethernet address: 00:1b:21:3d:eb:e7
> atapci0: <ATI AHCI controller> port
> 0xb000-0xb007,0xa000-0xa003,0x9000-0x9007,0x8000-0x8003,0x7000-0x700f
> mem 0xfbaffc00-0xfbafffff irq 22 at device 17.0 on pci0
> atapci0: [ITHREAD]
> atapci0: AHCI Version 01.10 controller with 6 ports detected
> ata2: <ATA channel 0> on atapci0
> ata2: [ITHREAD]
> ata3: <ATA channel 1> on atapci0
> ata3: [ITHREAD]
> ata4: <ATA channel 2> on atapci0
> ata4: [ITHREAD]
> ata5: <ATA channel 3> on atapci0
> ata5: [ITHREAD]
> ata6: <ATA channel 4> on atapci0
> ata6: [ITHREAD]
> ata7: <ATA channel 5> on atapci0
> ata7: [ITHREAD]
> ohci0: <OHCI (generic) USB controller> mem 0xfbafd000-0xfbafdfff irq 16
> at device 18.0 on pci0
> ohci0: [GIANT-LOCKED]
> ohci0: [ITHREAD]
> usb0: OHCI version 1.0, legacy support
> usb0: <OHCI (generic) USB controller> on ohci0
> usb0: USB revision 1.0
> uhub0: <ATI OHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb0
> uhub0: 3 ports with 3 removable, self powered
> ohci1: <OHCI (generic) USB controller> mem 0xfbafe000-0xfbafefff irq 16
> at device 18.1 on pci0
> ohci1: [GIANT-LOCKED]
> ohci1: [ITHREAD]
> usb1: OHCI version 1.0, legacy support
> usb1: <OHCI (generic) USB controller> on ohci1
> usb1: USB revision 1.0
> uhub1: <ATI OHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb1
> uhub1: 3 ports with 3 removable, self powered
> ehci0: <EHCI (generic) USB 2.0 controller> mem 0xfbaff800-0xfbaff8ff irq
> 17 at device 18.2 on pci0
> ehci0: [GIANT-LOCKED]
> ehci0: [ITHREAD]
> usb2: EHCI version 1.0
> usb2: companion controllers, 3 ports each: usb0 usb1
> usb2: <EHCI (generic) USB 2.0 controller> on ehci0
> usb2: USB revision 2.0
> uhub2: <ATI EHCI root hub, class 9/0, rev 2.00/1.00, addr 1> on usb2
> uhub2: 6 ports with 6 removable, self powered
> ohci2: <OHCI (generic) USB controller> mem 0xfbafb000-0xfbafbfff irq 18
> at device 19.0 on pci0
> ohci2: [GIANT-LOCKED]
> ohci2: [ITHREAD]
> usb3: OHCI version 1.0, legacy support
> usb3: <OHCI (generic) USB controller> on ohci2
> usb3: USB revision 1.0
> uhub3: <ATI OHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb3
> uhub3: 3 ports with 3 removable, self powered
> ohci3: <OHCI (generic) USB controller> mem 0xfbafc000-0xfbafcfff irq 18
> at device 19.1 on pci0
> ohci3: [GIANT-LOCKED]
> ohci3: [ITHREAD]
> usb4: OHCI version 1.0, legacy support
> usb4: <OHCI (generic) USB controller> on ohci3
> usb4: USB revision 1.0
> uhub4: <ATI OHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb4
> uhub4: 3 ports with 3 removable, self powered
> ehci1: <EHCI (generic) USB 2.0 controller> mem 0xfbaff400-0xfbaff4ff irq
> 19 at device 19.2 on pci0
> ehci1: [GIANT-LOCKED]
> ehci1: [ITHREAD]
> usb5: EHCI version 1.0
> usb5: companion controllers, 3 ports each: usb3 usb4
> usb5: <EHCI (generic) USB 2.0 controller> on ehci1
> usb5: USB revision 2.0
> uhub5: <ATI EHCI root hub, class 9/0, rev 2.00/1.00, addr 1> on usb5
> uhub5: 6 ports with 6 removable, self powered
> pci0: <serial bus, SMBus> at device 20.0 (no driver attached)
> atapci1: <ATI IXP700 UDMA133 controller> port
> 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xff00-0xff0f at device 20.1 on pci0
> ata0: <ATA channel 0> on atapci1
> ata0: [ITHREAD]
> hdac1: <ATI SB600 High Definition Audio Controller> mem
> 0xfbaf4000-0xfbaf7fff irq 16 at device 20.2 on pci0
> hdac1: HDA Driver Revision: 20090624_0136
> hdac1: [ITHREAD]
> isab0: <PCI-ISA bridge> at device 20.3 on pci0
> isa0: <ISA bus> on isab0
> pcib4: <ACPI PCI-PCI bridge> at device 20.4 on pci0
> pci4: <ACPI PCI bus> on pcib4
> ohci4: <OHCI (generic) USB controller> mem 0xfbafa000-0xfbafafff irq 18
> at device 20.5 on pci0
> ohci4: [GIANT-LOCKED]
> ohci4: [ITHREAD]
> usb6: OHCI version 1.0, legacy support
> usb6: <OHCI (generic) USB controller> on ohci4
> usb6: USB revision 1.0
> uhub6: <ATI OHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb6
> uhub6: 2 ports with 2 removable, self powered
> acpi_button0: <Power Button> on acpi0
> atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
> atkbd0: <AT Keyboard> irq 1 on atkbdc0
> kbd0 at atkbd0
> atkbd0: [GIANT-LOCKED]
> atkbd0: [ITHREAD]
> psm0: <PS/2 Mouse> irq 12 on atkbdc0
> psm0: [GIANT-LOCKED]
> psm0: [ITHREAD]
> psm0: model MouseMan+, device ID 0
> acpi_aiboost0: <ASUStek AIBOOSTER> on acpi0
> sio0: configured irq 4 not in bitmap of probed irqs 0
> sio0: port may not be enabled
> sio0: configured irq 4 not in bitmap of probed irqs 0
> sio0: port may not be enabled
> sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 flags 0x10 on
> acpi0
> sio0: type 16550A
> sio0: [FILTER]
> fdc0: <floppy drive controller (FDE)> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2
> on acpi0
> fdc0: [FILTER]
> cpu0: <ACPI CPU> on acpi0
> acpi_throttle0: <ACPI CPU Throttling> on cpu0
> cpu1: <ACPI CPU> on acpi0
> cpu2: <ACPI CPU> on acpi0
> cpu3: <ACPI CPU> on acpi0
> cryptosoft0: <software crypto> on motherboard
> orm0: <ISA Option ROM> at iomem 0xd7800-0xd87ff on isa0
> sc0: <System console> at flags 0x100 on isa0
> sc0: VGA <16 virtual consoles, flags=0x300>
> vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
> ppc0: cannot reserve I/O port range
> sio1: configured irq 3 not in bitmap of probed irqs 0
> sio1: port may not be enabled
> ulpt0: <Samsung Electronics Co., Ltd. Samsung Laser Printer ML-1200
> Series, class 0/0, rev 1.10/1.00, addr 2> on uhub0
> ulpt0: using bi-directional mode
> WARNING: ZFS is considered to be an experimental feature in FreeBSD.
> Timecounters tick every 1.000 msec
> vboxdrv: fAsync=0 offMin=0x12c offMax=0x219
> hptrr: start channel [0,0]
> hptrr: start channel [0,1]
> hptrr: start channel [0,2]
> hptrr: start channel [0,3]
> hptrr: [0 0] Start channel soft reset.
> hptrr: [0 2] Start channel soft reset.
> hptrr: [0 3] Start channel soft reset.
> hptrr: channel [0,0] started successfully
> hptrr: channel [0,2] started successfully
> hptrr: channel [0,3] started successfully
> hptrr: [0 1] Failed to perform channel hard reset.
> hptrr0: [GIANT-LOCKED]
> hptrr0: [ITHREAD]
> The GEOM class LABEL is already loaded.
> ZFS filesystem version 13
> ZFS storage pool version 13
> da0 at hptrr0 bus 0 target 0 lun 0
> da0: <HPT DISK 0_0 4.00> Fixed Direct Access SCSI-0 device
> da1 at hptrr0 bus 0 target 1 lun 0
> da1: <HPT DISK 0_1 4.00> Fixed Direct Access SCSI-0 device
> da2 at hptrr0 bus 0 target 2 lun 0
> da2: <HPT DISK 0_2 4.00> Fixed Direct Access SCSI-0 device
> acd0: DVDR <HL-DT-ST DVDRAM GSA-4163B/A104> at ata0-master PIO4
> ad4: 476940MB <WDC WD5000AAKS-00C8A0 12.01C02> at ata2-master SATA300
> acd0: FAILURE - INQUIRY ILLEGAL REQUEST asc=0x24 ascq=0x00 sks=0x40 0x00
> 0x01
> (probe0:ata0:0:0:0): TEST UNIT READY. CDB: 0 0 0 0 0 0
> (probe0:ata0:0:0:0): CAM Status: SCSI Status Error
> (probe0:ata0:0:0:0): SCSI Status: Check Condition
> (probe0:ata0:0:0:0): NOT READY csi:0,0,bb,0 asc:3a,0
> (probe0:ata0:0:0:0): Medium not present
> (probe0:ata0:0:0:0): Unretryable error
> cd0 at ata0 bus 0 target 0 lun 0
> cd0: <HL-DT-ST DVDRAM GSA-4163B A104> Removable CD-ROM SCSI-0 device
> cd0: 16.000MB/s transfers
> cd0: Attempt to query device size failed: NOT READY, Medium not present
> ad6: 953869MB <WDC WD10EADS-00L5B1 01.01A01> at ata3-master SATA300
> ad10: 953869MB <WDC WD10EACS-00D6B0 01.01A01> at ata5-master SATA300
> ad12: 476940MB <WDC WD5000AACS-00ZUB0 01.01B01> at ata6-master SATA300
> ad14: 953869MB <Seagate ST31000528AS CC44> at ata7-master SATA300
> hdac0: HDA Codec #0: ATI RS690/780 HDMI
> pcm0: <HDA ATI RS690/780 HDMI PCM #0 Digital> at cad 0 nid 1 on hdac0
> hdac1: HDA Codec #0: VIA VT1708S_0
> pcm1: <HDA VIA VT1708S_0 PCM #0 Analog> at cad 0 nid 1 on hdac1
> pcm2: <HDA VIA VT1708S_0 PCM #1 Digital> at cad 0 nid 1 on hdac1
> pcm3: <HDA VIA VT1708S_0 PCM #2 Digital> at cad 0 nid 1 on hdac1
> SMP: AP CPU #2 Launched!
> SMP: AP CPU #3 Launched!
> SMP: AP CPU #1 Launched!
> Trying to mount root from zfs:rootzfs
> <118>Loading configuration files.
> <118>kernel dumps on /dev/ad4s1b
> <118>Entropy harvesting:
> <118> interrupts
> <118> ethernet
> <118> point_to_point
> <118> kickstart
> <118>.
> GEOM_ELI: Device label/swapB.eli created.
> GEOM_ELI: Encryption: AES-CBC 256
> GEOM_ELI:     Crypto: software
> <118>swapon: adding /dev/label/swapB.eli as swap device
> <118>/etc/rc: WARNING: Use of the early.sh script is deprecated
> <118>/etc/rc: WARNING: Please use a new-style rc.d script instead
> <118>/etc/rc: WARNING: See rc(8) for more information
> <118>gmirror:
> <118>No such device: swap_AB.
> <118>
> <118>Starting file system checks:
> <118>/dev/label/bstrapA: FILE SYSTEM CLEAN; SKIPPING CHECKS
> <118>/dev/label/bstrapA: clean, 303581 free (261 frags, 37915 blocks,
> 0.1% fragmentation)
> <118>/dev/label/bstrapB: FILE SYSTEM CLEAN; SKIPPING CHECKS
> <118>/dev/label/bstrapB: clean, 303502 free (318 frags, 37898 blocks,
> 0.1% fragmentation)
> <118>Setting hostuuid: e039c626-5f54-da11-8e5a-e6b6a8a23cf0.
> <118>Setting hostid: 0xd449202f.
> <118>Mounting local file systems:
> <118>.
>
>
> Fatal trap 12: page fault while in kernel mode
> cpuid = 1; apic id = 01
> fault virtual address    = 0x50
> fault code        = supervisor read data, page not present
> instruction pointer    = 0x8:0xffffffff80ad7a5e
> stack pointer            = 0x10:0xffffff80a5b2f620
> frame pointer            = 0x10:0xffffff80a5b2f6a0
> code segment        = base 0x0, limit 0xfffff, type 0x1b
>            = DPL 0, pres 1, long 1, def32 0, gran 1
> processor eflags    = interrupt enabled, resume, IOPL = 0
> current process        = 314 (txg_thread_enter)
> trap number        = 12
> panic: page fault
> cpuid = 1
> Uptime: 2m11s
> Physical memory: 8050 MB
> Dumping 1553 MB: 1538 1522 1506 1490 1474 1458 1442 1426 1410 1394 1378
> 1362 1346 1330 1314 1298 1282 1266 1250 1234 1218 1202 1186 1170 1154
> 1138 1122 1106 1090 1074 1058 1042 1026 1010 994 978 962 946 930 914 898
> 882 866 850 834 818 802 786 770 754 738 722 706 690 674 658 642 626 610
> 594 578 562 546 530 514 498 482 466 450 434 418 402 386 370 354 338 322
> 306 290 274 258 242 226 210 194 178 162 146 130 114 98 82 66 50 34 18 2
>
> Reading symbols from /boot/kernel/zfs.ko...Reading symbols from
> /bootdirA/boot/kernel/zfs.ko.symbols...done.
> done.
> Loaded symbols for /boot/kernel/zfs.ko
> Reading symbols from /boot/kernel/opensolaris.ko...Reading symbols from
> /bootdirA/boot/kernel/opensolaris.ko.symbols...done.
> done.
> Loaded symbols for /boot/kernel/opensolaris.ko
> Reading symbols from /boot/kernel/geom_eli.ko...Reading symbols from
> /bootdirA/boot/kernel/geom_eli.ko.symbols...done.
> done.
> Loaded symbols for /boot/kernel/geom_eli.ko
> Reading symbols from /boot/kernel/crypto.ko...Reading symbols from
> /bootdirA/boot/kernel/crypto.ko.symbols...done.
> done.
> Loaded symbols for /boot/kernel/crypto.ko
> Reading symbols from /boot/kernel/zlib.ko...Reading symbols from
> /bootdirA/boot/kernel/zlib.ko.symbols...done.
> done.
> Loaded symbols for /boot/kernel/zlib.ko
> Reading symbols from /boot/kernel/geom_label.ko...Reading symbols from
> /bootdirA/boot/kernel/geom_label.ko.symbols...done.
> done.
> Loaded symbols for /boot/kernel/geom_label.ko
> Reading symbols from /boot/kernel/geom_mirror.ko...Reading symbols from
> /bootdirA/boot/kernel/geom_mirror.ko.symbols...done.
> done.
> Loaded symbols for /boot/kernel/geom_mirror.ko
> Reading symbols from /boot/kernel/snd_hda.ko...Reading symbols from
> /bootdirA/boot/kernel/snd_hda.ko.symbols...done.
> done.
> Loaded symbols for /boot/kernel/snd_hda.ko
> Reading symbols from /boot/kernel/sound.ko...Reading symbols from
> /bootdirA/boot/kernel/sound.ko.symbols...done.
> done.
> Loaded symbols for /boot/kernel/sound.ko
> Reading symbols from /boot/kernel/cd9660_iconv.ko...Reading symbols from
> /bootdirA/boot/kernel/cd9660_iconv.ko.symbols...done.
> done.
> Loaded symbols for /boot/kernel/cd9660_iconv.ko
> Reading symbols from /boot/kernel/libiconv.ko...Reading symbols from
> /bootdirA/boot/kernel/libiconv.ko.symbols...done.
> done.
> Loaded symbols for /boot/kernel/libiconv.ko
> Reading symbols from /boot/kernel/aio.ko...Reading symbols from
> /bootdirA/boot/kernel/aio.ko.symbols...done.
> done.
> Loaded symbols for /boot/kernel/aio.ko
> Reading symbols from /boot/modules/vboxdrv.ko...done.
> Loaded symbols for /boot/modules/vboxdrv.ko
> Reading symbols from /boot/kernel/sem.ko...Reading symbols from
> /bootdirA/boot/kernel/sem.ko.symbols...done.
> done.
> Loaded symbols for /boot/kernel/sem.ko
> #0  doadump () at pcpu.h:195
> 195        __asm __volatile("movq %%gs:0,%0" : "=r" (td));
> (kgdb) backtrace
> #0  doadump () at pcpu.h:195
> #1  0x0000000000000004 in ?? ()
> #2  0xffffffff803aab90 in boot (howto=260)
>    at /usr/src/sys/kern/kern_shutdown.c:418
> #3  0xffffffff803aafb2 in panic (fmt=0x104 <Address 0x104 out of bounds>)
>    at /usr/src/sys/kern/kern_shutdown.c:574
> #4  0xffffffff80651d03 in trap_fatal (frame=0xffffff0009572390,
> eva=Variable "eva" is not available.
> )
>    at /usr/src/sys/amd64/amd64/trap.c:770
> #5  0xffffffff806520d5 in trap_pfault (frame=0xffffff80a5b2f570, 
> usermode=0)
>    at /usr/src/sys/amd64/amd64/trap.c:686
> #6  0xffffffff80652a63 in trap (frame=0xffffff80a5b2f570)
>    at /usr/src/sys/amd64/amd64/trap.c:457
> #7  0xffffffff8063cb4e in calltrap ()
>    at /usr/src/sys/amd64/amd64/exception.S:218
> #8  0xffffffff80ad7a5e in find_block (th=0xffffff8006930000,
>    zseg=0xffffff000dafc580, dnp=0xffffff8006d35600, depth=Variable
> "depth" is not available.
> )
>    at
> /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_traverse.c:400 
>
> #9  0xffffffff80ad81a3 in traverse_more (th=0xffffff8006930000)
>    at
> /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_traverse.c:672 
>
> #10 0xffffffff80ad8368 in traverse_dsl_dataset (ds=0x23, txg_start=1,
> advance=Variable "advance" is not available.
>
> )
>    at
> /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/d---Type 
>
> <return> to continue, or q <return> to quit---
> mu_traverse.c:731
> #11 0xffffffff80ae41c9 in dsl_dataset_destroy_sync () from
> /boot/kernel/zfs.ko
> #12 0xffffffff80ae6d2a in dsl_sync_task_group_sync
> (dstg=0xffffff000d256600,
>    tx=0xffffff000d1fca80)
>    at
> /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_synctask.c:186 
>
> #13 0xffffffff80ae6873 in dsl_pool_sync (dp=0xffffff000d521000, 
> txg=3430336)
>    at
> /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_pool.c:316 
>
> #14 0xffffffff80af59c3 in spa_sync (spa=0xffffff0006d08000, txg=3430336)
>    at
> /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/spa.c:3988 
>
> #15 0xffffffff80afd772 in txg_sync_thread (arg=Variable "arg" is not
> available.
> )
>    at
> /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/txg.c:352 
>
> #16 0xffffffff80385452 in fork_exit (
>    callout=0xffffffff80afd490 <txg_sync_thread>, arg=0xffffff000d521000,
>    frame=0xffffff80a5b2fc80) at /usr/src/sys/kern/kern_fork.c:811
> #17 0xffffffff8063cf2e in fork_trampoline ()
>    at /usr/src/sys/amd64/amd64/exception.S:554
> #18 0x0000000000000000 in ?? ()
> #19 0x0000000000000000 in ?? ()
> #20 0x0000000000000001 in ?? ()
> #21 0x0000000000000000 in ?? ()
> #22 0x0000000000000000 in ?? ()
> #23 0x0000000000000000 in ?? ()
> #24 0x0000000000000000 in ?? ()
> #25 0x0000000000000000 in ?? ()
> #26 0x0000000000000000 in ?? ()
> #27 0x0000000000000000 in ?? ()
> #28 0x0000000000000000 in ?? ()
> #29 0x0000000000000000 in ?? ()
> #30 0x0000000000000000 in ?? ()
> #31 0x0000000000000000 in ?? ()
> #32 0x0000000000000000 in ?? ()
> #33 0x0000000000000000 in ?? ()
> #34 0x0000000000000000 in ?? ()
> #35 0x0000000000000000 in ?? ()
> #36 0x0000000000000000 in ?? ()
> #37 0x0000000000000000 in ?? ()
> #38 0x0000000000000000 in ?? ()
> #39 0x0000000000000000 in ?? ()
> #40 0x0000000000000000 in ?? ()
> #41 0x0000000000000000 in ?? ()
> #42 0x0000000000dde000 in ?? ()
> #43 0xffffffff80950d40 in tdq_cpu ()
> #44 0xffffffff8095c940 in tdq_groups ()
> #45 0xffffffff8095c8c0 in tdq_cpu ()
> #46 0xffffff0009572390 in ?? ()
> #47 0x0000000000000000 in ?? ()
> #48 0xffffff80a5b2f1e8 in ?? ()
> #49 0xffffff0009572390 in ?? ()
> #50 0xffffffff803ce154 in sched_switch (td=0xffffffff80afd490, newtd=0x0,
>    flags=0) at /usr/src/sys/kern/sched_ule.c:1938
> #51 0x0000000000000000 in ?? ()
> #52 0x0000000000000000 in ?? ()
> #53 0x0000000000000000 in ?? ()
> #54 0x0000000000000000 in ?? ()
> #55 0x0000000000000000 in ?? ()
> #56 0x0000000000000000 in ?? ()
> #57 0x0000000000000000 in ?? ()
> #58 0x0000000000000000 in ?? ()
> #59 0x0000000000000000 in ?? ()
> #60 0x0000000000000000 in ?? ()
> #61 0x0000000000000000 in ?? ()
> #62 0x0000000000000000 in ?? ()
> #63 0x0000000000000000 in ?? ()
> #64 0x0000000000000000 in ?? ()
> #65 0x0000000000000000 in ?? ()
> #66 0x0000000000000000 in ?? ()
> #67 0x0000000000000000 in ?? ()
> #68 0x0000000000000000 in ?? ()
> #69 0x0000000000000000 in ?? ()
> #70 0x0000000000000000 in ?? ()
> #71 0x0000000000000000 in ?? ()
> #72 0x0000000000000000 in ?? ()
> #73 0x0000000000000000 in ?? ()
> #74 0x0000000000000000 in ?? ()
> #75 0x0000000000000000 in ?? ()
> #76 0x0000000000000000 in ?? ()
> #77 0x0000000000000000 in ?? ()
> #78 0x0000000000000000 in ?? ()
> #79 0x0000000000000000 in ?? ()
> #80 0x0000000000000000 in ?? ()
> #81 0x0000000000000000 in ?? ()
> #82 0x0000000000000000 in ?? ()
> #83 0x0000000000000000 in ?? ()
> #84 0x0000000000000000 in ?? ()
> #85 0x0000000000000000 in ?? ()
> #86 0x0000000000000000 in ?? ()
> #87 0x0000000000000000 in ?? ()
> #88 0x0000000000000000 in ?? ()
> #89 0x0000000000000000 in ?? ()
> #90 0x0000000000000000 in ?? ()
> #91 0x0000000000000000 in ?? ()
> #92 0x0000000000000000 in ?? ()
> #93 0x0000000000000000 in ?? ()
> #94 0x0000000000000000 in ?? ()
> #95 0x0000000000000000 in ?? ()
> #96 0x0000000000000000 in ?? ()
> #97 0x0000000000000000 in ?? ()
> #98 0x0000000000000000 in ?? ()
> #99 0x0000000000000000 in ?? ()
> #100 0x0000000000000000 in ?? ()
> #101 0x0000000000000000 in ?? ()
> #102 0x0000000000000000 in ?? ()
> #103 0x0000000000000000 in ?? ()
> #104 0x0000000000000000 in ?? ()
> #105 0x0000000000000000 in ?? ()
> #106 0x0000000000000000 in ?? ()
> #107 0x0000000000000000 in ?? ()
> #108 0x0000000000000000 in ?? ()
> #109 0x0000000000000000 in ?? ()
> #110 0x0000000000000000 in ?? ()
> #111 0x0000000000000000 in ?? ()
> #112 0x0000000000000000 in ?? ()
> #113 0x0000000000000000 in ?? ()
> #114 0x0000000000000000 in ?? ()
> #115 0x0000000000000000 in ?? ()
> #116 0x0000000000000000 in ?? ()
> #117 0x0000000000000000 in ?? ()
> #118 0x0000000000000000 in ?? ()
> Cannot access memory at address 0xffffff80a5b30000
> (kgdb) q
> # ^D
> Script done on Fri Sep 11 17:26:02 2009
>
> _______________________________________________
> freebsd-current_at_freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-current
> To unsubscribe, send any mail to 
> "freebsd-current-unsubscribe_at_freebsd.org"
Is the hardware in the machine (CPU, motherboard, and memory) confirmed 
to be solid? I ran into a similar situation last month 
(http://lists.freebsd.org/pipermail/freebsd-fs/2009-August/006606.html), 
the root cause of which turned out to be bad memory. ZFS tends to make 
the problem show up more frequently since it uses a lot of memory.

-Boris