[ZFS][PANIC] Solaris Assert/zio.c:2548

From: Larry Rosenman <ler_at_lerctr.org>
Date: Sun, 20 Jul 2014 09:03:50 -0500
Got the following panic overnight (I think while a nightly rsync was running):

Dump header from device /dev/gpt/swap0
  Architecture: amd64
  Architecture Version: 2
  Dump Length: 8122101760B (7745 MB)
  Blocksize: 512
  Dumptime: Sun Jul 20 03:22:18 2014
  Hostname: borg.lerctr.org
  Magic: FreeBSD Kernel Dump
  Version String: FreeBSD 11.0-CURRENT #50 r268894M: Sat Jul 19 18:06:08 CDT 2014
    root_at_borg.lerctr.org:/usr/obj/usr/src/sys/VT-LER
  Panic String: solaris assert: !(zio->io_flags & ZIO_FLAG_DELEGATED), file: /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c, line: 2874
  Dump Parity: 763150733
  Bounds: 5
  Dump Status: good


borg.lerctr.org dumped core - see /var/crash/vmcore.5

Sun Jul 20 03:28:12 CDT 2014

FreeBSD borg.lerctr.org 11.0-CURRENT FreeBSD 11.0-CURRENT #50 r268894M: Sat Jul 19 18:06:08 CDT 2014     root_at_borg.lerctr.org:/usr/obj/usr/src/sys/VT-LER  amd64

panic: solaris assert: !(zio->io_flags & ZIO_FLAG_DELEGATED), file: /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c, line: 2874

GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd"...

Unread portion of the kernel message buffer:
panic: solaris assert: !(zio->io_flags & ZIO_FLAG_DELEGATED), file: /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c, line: 2874
cpuid = 7
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe100c49f930
kdb_backtrace() at kdb_backtrace+0x39/frame 0xfffffe100c49f9e0
vpanic() at vpanic+0x126/frame 0xfffffe100c49fa20
panic() at panic+0x43/frame 0xfffffe100c49fa80
assfail() at assfail+0x1d/frame 0xfffffe100c49fa90
zio_vdev_io_assess() at zio_vdev_io_assess+0x2ed/frame 0xfffffe100c49fac0
zio_execute() at zio_execute+0x1e9/frame 0xfffffe100c49fb20
taskqueue_run_locked() at taskqueue_run_locked+0xf0/frame 0xfffffe100c49fb80
taskqueue_thread_loop() at taskqueue_thread_loop+0x9b/frame 0xfffffe100c49fbb0
fork_exit() at fork_exit+0x84/frame 0xfffffe100c49fbf0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe100c49fbf0
--- trap 0, rip = 0, rsp = 0xfffffe100c49fcb0, rbp = 0 ---
Uptime: 8h57m17s
(ada2:ahcich2:0:0:0): FLUSHCACHE48. ACB: ea 00 00 00 00 40 00 00 00 00 00 00
(ada2:ahcich2:0:0:0): CAM status: Command timeout
(ada2:ahcich2:0:0:0): Error 5, Retries exhausted
(ada2:ahcich2:0:0:0): Synchronize cache failed
Dumping 7745 out of 64463 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91%

Reading symbols from /boot/kernel/linux.ko.symbols...done.
Loaded symbols for /boot/kernel/linux.ko.symbols
Reading symbols from /boot/kernel/if_lagg.ko.symbols...done.
Loaded symbols for /boot/kernel/if_lagg.ko.symbols
Reading symbols from /boot/kernel/snd_envy24ht.ko.symbols...done.
Loaded symbols for /boot/kernel/snd_envy24ht.ko.symbols
Reading symbols from /boot/kernel/snd_spicds.ko.symbols...done.
Loaded symbols for /boot/kernel/snd_spicds.ko.symbols
Reading symbols from /boot/kernel/coretemp.ko.symbols...done.
Loaded symbols for /boot/kernel/coretemp.ko.symbols
Reading symbols from /boot/kernel/ichsmb.ko.symbols...done.
Loaded symbols for /boot/kernel/ichsmb.ko.symbols
Reading symbols from /boot/kernel/smbus.ko.symbols...done.
Loaded symbols for /boot/kernel/smbus.ko.symbols
Reading symbols from /boot/kernel/ichwd.ko.symbols...done.
Loaded symbols for /boot/kernel/ichwd.ko.symbols
Reading symbols from /boot/kernel/cpuctl.ko.symbols...done.
Loaded symbols for /boot/kernel/cpuctl.ko.symbols
Reading symbols from /boot/kernel/crypto.ko.symbols...done.
Loaded symbols for /boot/kernel/crypto.ko.symbols
Reading symbols from /boot/kernel/cryptodev.ko.symbols...done.
Loaded symbols for /boot/kernel/cryptodev.ko.symbols
Reading symbols from /boot/kernel/dtraceall.ko.symbols...done.
Loaded symbols for /boot/kernel/dtraceall.ko.symbols
Reading symbols from /boot/kernel/profile.ko.symbols...done.
Loaded symbols for /boot/kernel/profile.ko.symbols
Reading symbols from /boot/kernel/cyclic.ko.symbols...done.
Loaded symbols for /boot/kernel/cyclic.ko.symbols
Reading symbols from /boot/kernel/dtrace.ko.symbols...done.
Loaded symbols for /boot/kernel/dtrace.ko.symbols
Reading symbols from /boot/kernel/systrace_freebsd32.ko.symbols...done.
Loaded symbols for /boot/kernel/systrace_freebsd32.ko.symbols
Reading symbols from /boot/kernel/systrace.ko.symbols...done.
Loaded symbols for /boot/kernel/systrace.ko.symbols
Reading symbols from /boot/kernel/sdt.ko.symbols...done.
Loaded symbols for /boot/kernel/sdt.ko.symbols
Reading symbols from /boot/kernel/lockstat.ko.symbols...done.
Loaded symbols for /boot/kernel/lockstat.ko.symbols
Reading symbols from /boot/kernel/fasttrap.ko.symbols...done.
Loaded symbols for /boot/kernel/fasttrap.ko.symbols
Reading symbols from /boot/kernel/fbt.ko.symbols...done.
Loaded symbols for /boot/kernel/fbt.ko.symbols
Reading symbols from /boot/kernel/dtnfscl.ko.symbols...done.
Loaded symbols for /boot/kernel/dtnfscl.ko.symbols
Reading symbols from /boot/kernel/dtmalloc.ko.symbols...done.
Loaded symbols for /boot/kernel/dtmalloc.ko.symbols
Reading symbols from /boot/modules/vboxdrv.ko...done.
Loaded symbols for /boot/modules/vboxdrv.ko
Reading symbols from /boot/kernel/ipmi.ko.symbols...done.
Loaded symbols for /boot/kernel/ipmi.ko.symbols
Reading symbols from /boot/kernel/ipmi_linux.ko.symbols...done.
Loaded symbols for /boot/kernel/ipmi_linux.ko.symbols
Reading symbols from /boot/kernel/radeonkms.ko.symbols...done.
Loaded symbols for /boot/kernel/radeonkms.ko.symbols
Reading symbols from /boot/kernel/iicbb.ko.symbols...done.
Loaded symbols for /boot/kernel/iicbb.ko.symbols
Reading symbols from /boot/kernel/iicbus.ko.symbols...done.
Loaded symbols for /boot/kernel/iicbus.ko.symbols
Reading symbols from /boot/kernel/iic.ko.symbols...done.
Loaded symbols for /boot/kernel/iic.ko.symbols
Reading symbols from /boot/kernel/drm2.ko.symbols...done.
Loaded symbols for /boot/kernel/drm2.ko.symbols
Reading symbols from /boot/kernel/radeonkmsfw_R100_cp.ko.symbols...done.
Loaded symbols for /boot/kernel/radeonkmsfw_R100_cp.ko.symbols
Reading symbols from /boot/kernel/fdescfs.ko.symbols...done.
Loaded symbols for /boot/kernel/fdescfs.ko.symbols
Reading symbols from /boot/kernel/linprocfs.ko.symbols...done.
Loaded symbols for /boot/kernel/linprocfs.ko.symbols
Reading symbols from /boot/kernel/uhid.ko.symbols...done.
Loaded symbols for /boot/kernel/uhid.ko.symbols
Reading symbols from /boot/modules/vboxnetflt.ko...done.
Loaded symbols for /boot/modules/vboxnetflt.ko
Reading symbols from /boot/kernel/netgraph.ko.symbols...done.
Loaded symbols for /boot/kernel/netgraph.ko.symbols
Reading symbols from /boot/kernel/ng_ether.ko.symbols...done.
Loaded symbols for /boot/kernel/ng_ether.ko.symbols
Reading symbols from /boot/modules/vboxnetadp.ko...done.
Loaded symbols for /boot/modules/vboxnetadp.ko
#0  doadump (textdump=1) at pcpu.h:219
219	pcpu.h: No such file or directory.
	in pcpu.h
(kgdb) #0  doadump (textdump=1) at pcpu.h:219
#1  0xffffffff80a055f7 in kern_reboot (howto=260)
    at /usr/src/sys/kern/kern_shutdown.c:445
#2  0xffffffff80a05b35 in vpanic (fmt=<value optimized out>, 
    ap=<value optimized out>) at /usr/src/sys/kern/kern_shutdown.c:744
#3  0xffffffff80a05b83 in panic (fmt=0x0)
    at /usr/src/sys/kern/kern_shutdown.c:673
#4  0xffffffff8032d05d in assfail (a=<value optimized out>, 
    f=<value optimized out>, l=<value optimized out>)
    at /usr/src/sys/cddl/compat/opensolaris/kern/opensolaris_cmn_err.c:81
#5  0xffffffff8040ad6d in zio_vdev_io_assess (ziop=<value optimized out>)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2874
#6  0xffffffff80405dd9 in zio_execute (zio=0xfffff809e449b730)
    at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1416
#7  0xffffffff80a4de60 in taskqueue_run_locked (queue=0xfffff8002255e800)
    at /usr/src/sys/kern/subr_taskqueue.c:356
#8  0xffffffff80a4e8db in taskqueue_thread_loop (arg=<value optimized out>)
    at /usr/src/sys/kern/subr_taskqueue.c:623
#9  0xffffffff809d3cc4 in fork_exit (
    callout=0xffffffff80a4e840 <taskqueue_thread_loop>, 
    arg=0xfffff80022611470, frame=0xfffffe100c49fc00)
    at /usr/src/sys/kern/kern_fork.c:977
#10 0xffffffff80df5afe in fork_trampoline ()
    at /usr/src/sys/amd64/amd64/exception.S:605
#11 0x0000000000000000 in ?? ()
Current language:  auto; currently minimal
(kgdb) 


vmcore is available.

I re-ran the rsync, and no panic.

Ideas?


-- 
Larry Rosenman                     http://www.lerctr.org/~ler
Phone: +1 214-642-9640                 E-Mail: ler_at_lerctr.org
US Mail: 108 Turvey Cove, Hutto, TX 78634-5688
Received on Sun Jul 20 2014 - 12:04:05 UTC

This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:40:50 UTC