Solaris Assert: How to fix?

From: Larry Rosenman <ler_at_lerctr.org>
Date: Sat, 30 Aug 2014 12:29:58 -0500
I'm getting:

borg.lerctr.org dumped core - see /var/crash/vmcore.0

Fri Aug 29 16:16:35 CDT 2014

FreeBSD borg.lerctr.org 11.0-CURRENT FreeBSD 11.0-CURRENT #0 r270811: Fri Aug 29 09:21:58 CDT 2014     root_at_borg.lerctr.org:/usr/obj/usr/src/sys/VT-LER  amd64

panic: solaris assert: l->l_phys->l_hdr.lh_block_type == ((1ULL << 63) + 0) (0x0 == 0x8000000000000000), file: /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zap.c, line: 534

GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd"...

Unread portion of the kernel message buffer:
Copyright (c) 1992-2014 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
	The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 11.0-CURRENT #0 r270811: Fri Aug 29 09:21:58 CDT 2014
    root_at_borg.lerctr.org:/usr/obj/usr/src/sys/VT-LER amd64
FreeBSD clang version 3.4.1 (tags/RELEASE_34/dot1-final 208032) 20140512
VT: running with driver "vga".
info: [drm] Initialized drm 1.1.0 20060810
CPU: Intel(R) Xeon(R) CPU           E5410  _at_ 2.33GHz (2327.55-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x10676  Family=0x6  Model=0x17  Stepping=6
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0xce3bd<SSE3,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,SSE4.1>
  AMD Features=0x20100800<SYSCALL,NX,LM>
  AMD Features2=0x1<LAHF>
  VT-x: HLT,PAUSE
  TSC: P-state invariant, performance statistics
real memory  = 68719476736 (65536 MB)
avail memory = 65627250688 (62587 MB)
Event timer "LAPIC" quality 400
ACPI APIC Table: <PTLTD  	 APIC  >
FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs
FreeBSD/SMP: 2 package(s) x 4 core(s)
 cpu0 (BSP): APIC ID:  0
 cpu1 (AP): APIC ID:  1
 cpu2 (AP): APIC ID:  2
 cpu3 (AP): APIC ID:  3
 cpu4 (AP): APIC ID:  4
 cpu5 (AP): APIC ID:  5
 cpu6 (AP): APIC ID:  6
 cpu7 (AP): APIC ID:  7
ioapic0 <Version 2.0> irqs 0-23 on motherboard
ioapic1 <Version 2.0> irqs 24-47 on motherboard
kbd1 at kbdmux0
random: <Software, Yarrow> initialized
module_register_init: MOD_LOAD (vesa, 0xffffffff80e94c10, 0) error 19
cryptosoft0: <software crypto> on motherboard
acpi0: <PTLTD   RSDT> on motherboard
acpi0: Power Button (fixed)
unknown: I/O range not supported
cpu0: <ACPI CPU> on acpi0
cpu1: <ACPI CPU> on acpi0
cpu2: <ACPI CPU> on acpi0
cpu3: <ACPI CPU> on acpi0
cpu4: <ACPI CPU> on acpi0
cpu5: <ACPI CPU> on acpi0
cpu6: <ACPI CPU> on acpi0
cpu7: <ACPI CPU> on acpi0
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff irq 0,8 on acpi0
Timecounter "HPET" frequency 14318180 Hz quality 950
Event timer "HPET" frequency 14318180 Hz quality 350
Event timer "HPET1" frequency 14318180 Hz quality 340
Event timer "HPET2" frequency 14318180 Hz quality 340
atrtc0: <AT realtime clock> port 0x70-0x71 on acpi0
Event timer "RTC" frequency 32768 Hz quality 0
attimer0: <AT timer> port 0x40-0x43,0x50-0x53 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> at device 2.0 on pci0
pci1: <ACPI PCI bus> on pcib1
pcib2: <ACPI PCI-PCI bridge> irq 16 at device 0.0 on pci1
pci2: <ACPI PCI bus> on pcib2
pcib3: <ACPI PCI-PCI bridge> irq 16 at device 0.0 on pci2
pci3: <ACPI PCI bus> on pcib3
pcib4: <ACPI PCI-PCI bridge> at device 0.0 on pci3
pci4: <ACPI PCI bus> on pcib4
pcib5: <ACPI PCI-PCI bridge> at device 0.2 on pci3
pci5: <ACPI PCI bus> on pcib5
pcib6: <ACPI PCI-PCI bridge> irq 18 at device 2.0 on pci2
pci6: <ACPI PCI bus> on pcib6
em0: <Intel(R) PRO/1000 Network Connection 7.4.2> port 0x2000-0x201f mem 0xd9220000-0xd923ffff,0xd9200000-0xd921ffff irq 18 at device 0.0 on pci6
em0: Using an MSI interrupt
em0: Ethernet address: 00:30:48:f2:29:9c
em1: <Intel(R) PRO/1000 Network Connection 7.4.2> port 0x2020-0x203f mem 0xd9260000-0xd927ffff,0xd9240000-0xd925ffff irq 19 at device 0.1 on pci6
em1: Using an MSI interrupt
em1: Ethernet address: 00:30:48:f2:29:9d
pcib7: <ACPI PCI-PCI bridge> at device 0.3 on pci1
pci7: <ACPI PCI bus> on pcib7
pcib8: <ACPI PCI-PCI bridge> at device 4.0 on pci0
pci8: <ACPI PCI bus> on pcib8
vgapci0: <VGA-compatible display> port 0x3000-0x307f mem 0xd8000000-0xd8ffffff,0xc0000000-0xc7ffffff,0xc8000000-0xc9ffffff irq 16 at device 0.0 on pci8
nvidia0: <GeForce GT 610> on vgapci0
vgapci0: child nvidia0 requested pci_enable_io
vgapci0: child nvidia0 requested pci_enable_io
hdac0: <NVIDIA GF119 HDA Controller> mem 0xd9000000-0xd9003fff irq 17 at device 0.1 on pci8
pcib9: <ACPI PCI-PCI bridge> at device 6.0 on pci0
pci9: <ACPI PCI bus> on pcib9
pcib10: <ACPI PCI-PCI bridge> irq 17 at device 28.0 on pci0
pci10: <ACPI PCI bus> on pcib10
pcib11: <PCI-PCI bridge> irq 16 at device 0.0 on pci10
pci11: <PCI bus> on pcib11
pcm0: <Envy24HT audio (Generic)> port 0x4080-0x409f,0x4000-0x407f irq 16 at device 0.0 on pci11
pcm0: system configuration
  SubVendorID: 0x1412, SubDeviceID: 0x2403
  XIN2 Clock Source: 24.576MHz(96kHz*256)
  MPU-401 UART(s) #: not implemented
  ADC #: 1 and SPDIF receiver connected
  DAC #: 4
  Multi-track converter type: AC'97(SDATA_OUT:packed)
  S/PDIF(IN/OUT): 1/1 ID# 0x00
  GPIO(mask/dir/state): 0xff/0xff/0xff
uhci0: <Intel 631XESB/632XESB/3100 USB controller USB-1> port 0x1800-0x181f irq 17 at device 29.0 on pci0
usbus0 on uhci0
uhci1: <Intel 631XESB/632XESB/3100 USB controller USB-2> port 0x1820-0x183f irq 19 at device 29.1 on pci0
usbus1 on uhci1
uhci2: <Intel 631XESB/632XESB/3100 USB controller USB-3> port 0x1840-0x185f irq 18 at device 29.2 on pci0
usbus2 on uhci2
ehci0: <Intel 63XXESB USB 2.0 controller> mem 0xd9600400-0xd96007ff irq 17 at device 29.7 on pci0
usbus3: EHCI version 1.0
usbus3 on ehci0
pcib12: <ACPI PCI-PCI bridge> at device 30.0 on pci0
pci12: <ACPI PCI bus> on pcib12
vgapci1: <VGA-compatible display> port 0x5000-0x50ff mem 0xd0000000-0xd7ffffff,0xd9300000-0xd930ffff irq 18 at device 1.0 on pci12
drmn1: <ATI ES1000 RN50> on vgapci1
info: [drm] RADEON_IS_PCI
info: [drm] initializing kernel modesetting (RV100 0x1002:0x515E 0x15D9:0x8080).
info: [drm] register mmio base: 0xD9300000
info: [drm] register mmio size: 65536
info: [drm] radeon_atrm_get_bios: ===> Try ATRM...
info: [drm] radeon_atrm_get_bios: pci_find_class() found: 0:8:0:0, vendor=10de, device=104a
info: [drm] radeon_atrm_get_bios: Get ACPI device handle
info: [drm] radeon_acpi_vfct_bios: ===> Try VFCT...
info: [drm] radeon_acpi_vfct_bios: Get "VFCT" ACPI table
info: [drm] radeon_acpi_vfct_bios: Failed to get "VFCT" table: AE_NOT_FOUND
info: [drm] igp_read_bios_from_vram: ===> Try IGP's VRAM...
info: [drm] igp_read_bios_from_vram: VRAM base address: 0xd0000000
info: [drm] igp_read_bios_from_vram: Map address: 0xfffff800d0000000 (262144 bytes)
info: [drm] igp_read_bios_from_vram: Incorrect BIOS signature: 0x0000
info: [drm] radeon_read_bios: ===> Try PCI Expansion ROM...
info: [drm] radeon_read_bios: Map address: 0xfffff800000c0000 (131072 bytes)
drmn1: info: VRAM: 128M 0x00000000D0000000 - 0x00000000D7FFFFFF (16M used)
drmn1: info: GTT: 512M 0x00000000B0000000 - 0x00000000CFFFFFFF
info: [drm] Supports vblank timestamp caching Rev 1 (10.10.2010).
info: [drm] Driver supports precise vblank timestamp query.
info: [drm] radeon: irq initialized.
info: [drm] Detected VRAM RAM=128M, BAR=128M
info: [drm] RAM width 64bits SDR
[TTM] Zone  kernel: Available graphics memory: 33001600 kiB
[TTM] Zone   dma32: Available graphics memory: 2097152 kiB
[TTM] Initializing pool allocator
info: [drm] radeon: 16M of VRAM memory ready
info: [drm] radeon: 512M of GTT memory ready.
info: [drm] GART: num cpu pages 131072, num gpu pages 131072
info: [drm] PCI GART of 512M enabled (table at 0x0000000010CF6000).
drmn1: info: WB disabled
drmn1: info: fence driver on ring 0 use gpu addr 0x00000000b0000000 and cpu addr 0x0xfffff800118de000
info: [drm] Loading R100 Microcode
info: [drm] radeon: ring at 0x00000000B0001000
info: [drm] ring test succeeded in 1 usecs
info: [drm] ib test succeeded in 0 usecs
info: [drm] radeon_device_init: Taking over the fictitious range 0xd0000000-0xd4000000
iicbus0: <Philips I2C bus> on iicbb0 addr 0xff
iic0: <I2C generic I/O> on iicbus0
iicbus1: <Philips I2C bus> on iicbb1 addr 0x0
iic1: <I2C generic I/O> on iicbus1
iicbus2: <Philips I2C bus> on iicbb2 addr 0x0
iic2: <I2C generic I/O> on iicbus2
iicbus3: <Philips I2C bus> on iicbb3 addr 0x0
iic3: <I2C generic I/O> on iicbus3
info: [drm] Radeon Display Connectors
info: [drm] Connector 0:
info: [drm]   VGA-1
info: [drm]   DDC: 0x60 0x60 0x60 0x60 0x60 0x60 0x60 0x60
info: [drm]   Encoders:
info: [drm]     CRT1: INTERNAL_DAC1
info: [drm] Connector 1:
info: [drm]   DVI-I-1
info: [drm]   HPD2
info: [drm]   DDC: 0x6c 0x6c 0x6c 0x6c 0x6c 0x6c 0x6c 0x6c
info: [drm]   Encoders:
info: [drm]     CRT2: INTERNAL_DAC2
info: [drm]     DFP2: INTERNAL_DVO1
composite sync not supported
composite sync not supported
info: [drm] fb mappable at 0xD0040000
info: [drm] vram apper at 0xD0000000
info: [drm] size 2076672
info: [drm] fb depth is 8
info: [drm]    pitch is 1920
fbd1 on drmn1
VT: Replacing driver "vga" with new "fb".
info: [drm] Initialized radeon 2.29.0 20080528
vgapci1: Boot video device
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel 63XXESB2 UDMA100 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x1860-0x186f at device 31.1 on pci0
ata0: <ATA channel> at channel 0 on atapci0
ahci0: <Intel ESB2 AHCI SATA controller> port 0x18a0-0x18a7,0x1874-0x1877,0x1878-0x187f,0x1870-0x1873,0x1880-0x189f mem 0xd9600800-0xd9600bff irq 19 at device 31.2 on pci0
ahci0: AHCI v1.10 with 6 3Gbps ports, Port Multiplier supported
ahcich0: <AHCI channel> at channel 0 on ahci0
ahcich1: <AHCI channel> at channel 1 on ahci0
ahcich2: <AHCI channel> at channel 2 on ahci0
ahcich3: <AHCI channel> at channel 3 on ahci0
ahcich4: <AHCI channel> at channel 4 on ahci0
ahcich5: <AHCI channel> at channel 5 on ahci0
ichsmb0: <Intel 631xESB/6321ESB (ESB2) SMBus controller> port 0x1100-0x111f irq 19 at device 31.3 on pci0
smbus0: <System Management Bus> on ichsmb0
acpi_button0: <Power Button> on acpi0
ipmi0: <IPMI System Interface> port 0xca2-0xca3 on acpi0
ipmi0: KCS mode found at io 0xca2 on acpi
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0
fdc0: <floppy drive controller> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0
fd0: <1440-KB 3.5" drive> on fdc0 drive 0
ppc0: <Parallel port> port 0x378-0x37f,0x778-0x77f irq 7 drq 3 on acpi0
ppc0: SMC-like chipset (ECP/EPP/PS2/NIBBLE) in COMPATIBLE mode
ppc0: FIFO with 16/16/9 bytes threshold
ppbus0: <Parallel port bus> on ppc0
lpt0: <Printer> on ppbus0
lpt0: Interrupt-driven port
ppi0: <Parallel I/O> on ppbus0
ichwd0 on isa0
orm0: <ISA Option ROM> at iomem 0xc0000-0xcafff on isa0
coretemp0: <CPU On-Die Thermal Sensors> on cpu0
est0: <Enhanced SpeedStep Frequency Control> on cpu0
coretemp1: <CPU On-Die Thermal Sensors> on cpu1
est1: <Enhanced SpeedStep Frequency Control> on cpu1
coretemp2: <CPU On-Die Thermal Sensors> on cpu2
est2: <Enhanced SpeedStep Frequency Control> on cpu2
coretemp3: <CPU On-Die Thermal Sensors> on cpu3
est3: <Enhanced SpeedStep Frequency Control> on cpu3
coretemp4: <CPU On-Die Thermal Sensors> on cpu4
est4: <Enhanced SpeedStep Frequency Control> on cpu4
coretemp5: <CPU On-Die Thermal Sensors> on cpu5
est5: <Enhanced SpeedStep Frequency Control> on cpu5
coretemp6: <CPU On-Die Thermal Sensors> on cpu6
est6: <Enhanced SpeedStep Frequency Control> on cpu6
coretemp7: <CPU On-Die Thermal Sensors> on cpu7
est7: <Enhanced SpeedStep Frequency Control> on cpu7
random: unblocking device.
usbus0: 12Mbps Full Speed USB v1.0
ZFS filesystem version: 5
ZFS storage pool version: features support (5000)
Timecounters tick every 1.000 msec
vboxdrv: fAsync=0 offMin=0x444 offMax=0x690
hdacc0: <NVIDIA (0x001c) HDA CODEC> at cad 0 on hdac0
hdaa0: <NVIDIA (0x001c) Audio Function Group> at nid 1 on hdacc0
pcm1: <NVIDIA (0x001c) (HDMI/DP 8ch)> at nid 4 on hdaa0
pcm2: <NVIDIA (0x001c) (HDMI/DP 8ch)> at nid 5 on hdaa0
ugen0.1: <Intel> at usbus0
uhub0: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0
usbus1: 12Mbps Full Speed USB v1.0
usbus2: 12Mbps Full Speed USB v1.0
usbus3: 480Mbps High Speed USB v2.0
ugen3.1: <Intel> at usbus3
uhub1: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus3
ugen2.1: <Intel> at usbus2
uhub2: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus2
ugen1.1: <Intel> at usbus1
uhub3: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus1
ipmi0: IPMI device rev. 1, firmware rev. 1.64, version 2.0
ata0: DMA limited to UDMA33, controller found non-ATA66 cable
ipmi0: Number of channels 8
ipmi0: Attached watchdog
uhub0: 2 ports with 2 removable, self powered
uhub2: 2 ports with 2 removable, self powered
uhub3: 2 ports with 2 removable, self powered
ada0 at ahcich0 bus 0 scbus1 target 0 lun 0
ada0: <ST2000DL003-9VT166 CC32> ATA-8 SATA 3.x device
ada0: Serial Number 5YDA1ZL4
ada0: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada0: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
ada0: quirks=0x1<4K>
ada0: Previously was known as ad4
ada1 at ahcich1 bus 0 scbus2 target 0 lun 0
ada1: <ST2000DL003-9VT166 CC3D> ATA-8 SATA 3.x device
ada1: Serial Number 5YD6FPLG
ada1: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada1: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
ada1: quirks=0x1<4K>
ada1: Previously was known as ad6
ada2 at ahcich2 bus 0 scbus3 target 0 lun 0
ada2: <ST2000DL003-9VT166 CC32> ATA-8 SATA 3.x device
ada2: Serial Number 5YDA3PC5
ada2: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada2: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
ada2: quirks=0x1<4K>
ada2: Previously was known as ad8
ada3 at ahcich3 bus 0 scbus4 target 0 lun 0
ada3: <ST2000DL003-9VT166 CC32> ATA-8 SATA 3.x device
ada3: Serial Number 5YD9Y0P4
ada3: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada3: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
ada3: quirks=0x1<4K>
ada3: Previously was known as ad10
ada4 at ahcich4 bus 0 scbus5 target 0 lun 0
ada4: <ST2000DL003-9VT166 CC32> ATA-8 SATA 3.x device
ada4: Serial Number 5YDA1R5W
ada4: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada4: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
ada4: quirks=0x1<4K>
ada4: Previously was known as ad12
ada5 at ahcich5 bus 0 scbus6 target 0 lun 0
ada5: <ST2000DL003-9VT166 CC3D> ATA-8 SATA 3.x device
ada5: Serial Number 5YD5RBS8
ada5: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada5: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
ada5: quirks=0x1<4K>
ada5: Previously was known as ad14
cd0 at ata0 bus 0 scbus0 target 0 lun 0
cd0: <Memorex DVD+-RAM 510L v1 MWS7> Removable CD-ROM SCSI-0 device 
cd0: 33.300MB/s transfers (UDMA2, ATAPI 12bytes, PIO 65534bytes)
cd0: Attempt to query device size failed: NOT READY, Medium not present
SMP: AP CPU #4 Launched!
SMP: AP CPU #1 Launched!
SMP: AP CPU #3 Launched!
SMP: AP CPU #2 Launched!
SMP: AP CPU #5 Launched!
SMP: AP CPU #6 Launched!
SMP: AP CPU #7 Launched!
Root mount waiting for: usbus3
uhub1: 6 ports with 6 removable, self powered
Root mount waiting for: usbus3
Root mount waiting for: usbus3
ugen3.2: <Peppercon AG> at usbus3
ukbd0: <Peppercon AG Multidevice, class 0/0, rev 2.00/0.01, addr 2> on usbus3
kbd2 at ukbd0
Trying to mount root from zfs:zroot/ROOT/default []...
ugen0.2: <American Power Conversion> at usbus0
<118>Setting hostuuid: 53d19f64-d663-a017-8922-0030488e9ff3.
<118>Setting hostid: 0xf53a926e.
<118>Entropy harvesting: interrupts ethernet point_to_point swi.
<118>Starting file system checks:
<118>Mounting local file systems:.
panic: solaris assert: l->l_phys->l_hdr.lh_block_type == ((1ULL << 63) + 0) (0x0 == 0x8000000000000000), file: /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zap.c, line: 534
cpuid = 5
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe100be1d040
kdb_backtrace() at kdb_backtrace+0x39/frame 0xfffffe100be1d0f0
vpanic() at vpanic+0x189/frame 0xfffffe100be1d170
panic() at panic+0x43/frame 0xfffffe100be1d1d0
assfail3() at assfail3+0x2f/frame 0xfffffe100be1d1f0
zap_get_leaf_byblk() at zap_get_leaf_byblk+0x4e0/frame 0xfffffe100be1d240
zap_deref_leaf() at zap_deref_leaf+0xd3/frame 0xfffffe100be1d290
fzap_cursor_retrieve() at fzap_cursor_retrieve+0xf1/frame 0xfffffe100be1d300
zap_cursor_retrieve() at zap_cursor_retrieve+0x20d/frame 0xfffffe100be1d390
ddt_zap_walk() at ddt_zap_walk+0x53/frame 0xfffffe100be1d640
ddt_walk() at ddt_walk+0x79/frame 0xfffffe100be1d670
dsl_scan_sync() at dsl_scan_sync+0x8e7/frame 0xfffffe100be1d9d0
spa_sync() at spa_sync+0x5c1/frame 0xfffffe100be1dac0
txg_sync_thread() at txg_sync_thread+0x24d/frame 0xfffffe100be1dbb0
fork_exit() at fork_exit+0x84/frame 0xfffffe100be1dbf0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe100be1dbf0
--- trap 0, rip = 0, rsp = 0xfffffe100be1dcb0, rbp = 0 ---
Uptime: 14s
Dumping 2340 out of 64456 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91%

Reading symbols from /boot/kernel/linux.ko.symbols...done.
Loaded symbols for /boot/kernel/linux.ko.symbols
Reading symbols from /boot/kernel/if_lagg.ko.symbols...done.
Loaded symbols for /boot/kernel/if_lagg.ko.symbols
Reading symbols from /boot/kernel/snd_envy24ht.ko.symbols...done.
Loaded symbols for /boot/kernel/snd_envy24ht.ko.symbols
Reading symbols from /boot/kernel/snd_spicds.ko.symbols...done.
Loaded symbols for /boot/kernel/snd_spicds.ko.symbols
Reading symbols from /boot/kernel/coretemp.ko.symbols...done.
Loaded symbols for /boot/kernel/coretemp.ko.symbols
Reading symbols from /boot/kernel/ichsmb.ko.symbols...done.
Loaded symbols for /boot/kernel/ichsmb.ko.symbols
Reading symbols from /boot/kernel/smbus.ko.symbols...done.
Loaded symbols for /boot/kernel/smbus.ko.symbols
Reading symbols from /boot/kernel/ichwd.ko.symbols...done.
Loaded symbols for /boot/kernel/ichwd.ko.symbols
Reading symbols from /boot/kernel/cpuctl.ko.symbols...done.
Loaded symbols for /boot/kernel/cpuctl.ko.symbols
Reading symbols from /boot/kernel/crypto.ko.symbols...done.
Loaded symbols for /boot/kernel/crypto.ko.symbols
Reading symbols from /boot/kernel/cryptodev.ko.symbols...done.
Loaded symbols for /boot/kernel/cryptodev.ko.symbols
Reading symbols from /boot/kernel/dtraceall.ko.symbols...done.
Loaded symbols for /boot/kernel/dtraceall.ko.symbols
Reading symbols from /boot/kernel/profile.ko.symbols...done.
Loaded symbols for /boot/kernel/profile.ko.symbols
Reading symbols from /boot/kernel/cyclic.ko.symbols...done.
Loaded symbols for /boot/kernel/cyclic.ko.symbols
Reading symbols from /boot/kernel/dtrace.ko.symbols...done.
Loaded symbols for /boot/kernel/dtrace.ko.symbols
Reading symbols from /boot/kernel/systrace_freebsd32.ko.symbols...done.
Loaded symbols for /boot/kernel/systrace_freebsd32.ko.symbols
Reading symbols from /boot/kernel/systrace.ko.symbols...done.
Loaded symbols for /boot/kernel/systrace.ko.symbols
Reading symbols from /boot/kernel/sdt.ko.symbols...done.
Loaded symbols for /boot/kernel/sdt.ko.symbols
Reading symbols from /boot/kernel/lockstat.ko.symbols...done.
Loaded symbols for /boot/kernel/lockstat.ko.symbols
Reading symbols from /boot/kernel/fasttrap.ko.symbols...done.
Loaded symbols for /boot/kernel/fasttrap.ko.symbols
Reading symbols from /boot/kernel/fbt.ko.symbols...done.
Loaded symbols for /boot/kernel/fbt.ko.symbols
Reading symbols from /boot/kernel/dtnfscl.ko.symbols...done.
Loaded symbols for /boot/kernel/dtnfscl.ko.symbols
Reading symbols from /boot/kernel/dtmalloc.ko.symbols...done.
Loaded symbols for /boot/kernel/dtmalloc.ko.symbols
Reading symbols from /boot/modules/vboxdrv.ko...done.
Loaded symbols for /boot/modules/vboxdrv.ko
Reading symbols from /boot/modules/nvidia.ko...done.
Loaded symbols for /boot/modules/nvidia.ko
Reading symbols from /boot/kernel/ipmi.ko.symbols...done.
Loaded symbols for /boot/kernel/ipmi.ko.symbols
Reading symbols from /boot/kernel/ipmi_linux.ko.symbols...done.
Loaded symbols for /boot/kernel/ipmi_linux.ko.symbols
Reading symbols from /boot/kernel/radeonkms.ko.symbols...done.
Loaded symbols for /boot/kernel/radeonkms.ko.symbols
Reading symbols from /boot/kernel/iicbb.ko.symbols...done.
Loaded symbols for /boot/kernel/iicbb.ko.symbols
Reading symbols from /boot/kernel/iicbus.ko.symbols...done.
Loaded symbols for /boot/kernel/iicbus.ko.symbols
Reading symbols from /boot/kernel/iic.ko.symbols...done.
Loaded symbols for /boot/kernel/iic.ko.symbols
Reading symbols from /boot/kernel/drm2.ko.symbols...done.
Loaded symbols for /boot/kernel/drm2.ko.symbols
Reading symbols from /boot/kernel/radeonkmsfw_R100_cp.ko.symbols...done.
Loaded symbols for /boot/kernel/radeonkmsfw_R100_cp.ko.symbols
Reading symbols from /boot/kernel/fdescfs.ko.symbols...done.
Loaded symbols for /boot/kernel/fdescfs.ko.symbols
#0  doadump (textdump=1) at pcpu.h:219
219	pcpu.h: No such file or directory.
	in pcpu.h
(kgdb) #0  doadump (textdump=1) at pcpu.h:219
#1  0xffffffff80a15267 in kern_reboot (howto=260)
    at /usr/src/sys/kern/kern_shutdown.c:447
#2  0xffffffff80a15808 in vpanic (fmt=<value optimized out>, 
    ap=<value optimized out>) at /usr/src/sys/kern/kern_shutdown.c:746
#3  0xffffffff80a15853 in panic (fmt=0x0)
    at /usr/src/sys/kern/kern_shutdown.c:675
#4  0xffffffff8032f0af in assfail3 (a=<value optimized out>, 
    lv=<value optimized out>, op=<value optimized out>, 
    rv=<value optimized out>, f=<value optimized out>, 
    l=<value optimized out>)
    at /usr/src/sys/cddl/compat/opensolaris/kern/opensolaris_cmn_err.c:91
#5  0xffffffff803c99a0 in zap_get_leaf_byblk (zap=<value optimized out>, 
    blkid=<value optimized out>, tx=<value optimized out>, 
    lt=<value optimized out>, lp=0xfffffe100be1d4d8)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zap.c:534
#6  0xffffffff803c7053 in zap_deref_leaf (zap=0xfffff80024d3ff00, 
    h=9377620324092215296, tx=0x0, lt=RW_READER, lp=0xfffffe100be1d4d8)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zap.c:585
#7  0xffffffff803c8e11 in fzap_cursor_retrieve (zap=0xfffff80024d3ff00, 
    zc=0xfffffe100be1d4c8, za=0xfffffe100be1d3b0)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zap.c:1181
#8  0xffffffff803cf40d in zap_cursor_retrieve (zc=0xfffffe100be1d4c8, 
    za=0xfffffe100be1d3b0)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zap_micro.c:1290
#9  0xffffffff8035ca53 in ddt_zap_walk (os=0xfffff80024948000, object=156, 
    dde=0xfffffe100be1d818, walk=0xfffff8001181fad8)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/ddt_zap.c:117
#10 0xffffffff8035c709 in ddt_walk (spa=0xfffffe00110cd000, 
    ddb=0xfffff8001181fac0, dde=0xfffffe100be1d818)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/ddt.c:218
#11 0xffffffff8038c977 in dsl_scan_sync (dp=0xfffff80024095000, 
    tx=0xfffff800276b7600)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_scan.c:1194
#12 0xffffffff803a7a91 in spa_sync (spa=0xfffffe00110cd000, txg=17880931)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/spa.c:6610
#13 0xffffffff803b332d in txg_sync_thread (arg=0xfffff80024095000)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/txg.c:517
#14 0xffffffff809e3694 in fork_exit (
    callout=0xffffffff803b30e0 <txg_sync_thread>, arg=0xfffff80024095000, 
    frame=0xfffffe100be1dc00) at /usr/src/sys/kern/kern_fork.c:977
#15 0xffffffff80e07fce in fork_trampoline ()
    at /usr/src/sys/amd64/amd64/exception.S:605
#16 0x0000000000000000 in ?? ()
Current language:  auto; currently minimal
(kgdb) 


What can I do to fix the data on the disk?

-- 
Larry Rosenman                     http://www.lerctr.org/~ler
Phone: +1 214-642-9640                 E-Mail: ler_at_lerctr.org
US Mail: 108 Turvey Cove, Hutto, TX 78634-5688
Received on Sat Aug 30 2014 - 15:30:15 UTC

This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:40:51 UTC