Re: ZFS panic (dn->dn_datablkshift != 0) with r256304 and send/recv

on 14/10/2013 03:34 Keith White said the following:
> I get the following assert failure with a recent current:
> 
> panic: solaris assert: dn->dn_datablkshift != 0, file:
> /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_tx.c,
> line: 638

Please see https://www.illumos.org/issues/4188
The current best known fix is to simply drop the assertion.
Though, I am not entirely sure if this will be the final solution.  I'll
double-check with Matt.

> # uname -a
> FreeBSD freebsd10 11.0-CURRENT FreeBSD 11.0-CURRENT #0 r256304: Thu Oct 10
> 19:38:55 EDT 2013     kwhite_at_freebsd10:/tank/obj/usr/src/sys/GENERIC  amd64
> 
> # kgdb /boot/kernel/kernel /var/crash/vmcore.last
> GNU gdb 6.1.1 [FreeBSD]
> Copyright 2004 Free Software Foundation, Inc.
> GDB is free software, covered by the GNU General Public License, and you are
> welcome to change it and/or distribute copies of it under certain conditions.
> Type "show copying" to see the conditions.
> There is absolutely no warranty for GDB.  Type "show warranty" for details.
> This GDB was configured as "amd64-marcel-freebsd"...
> 
> Unread portion of the kernel message buffer:
> ...
> <118># zfs send -vi tank/RPI_at_20131004 tank/RPI_at_20131013 | zfs recv -vF
> m_tank/RPI_at_20131013
> <118>send from _at_20131004 to tank/RPI_at_20131013 estimated size is 85.0M
> <118>total estimated size is 85.0M
> <118>TIME        SENT   SNAPSHOT
> <118>receiving incremental stream of tank/RPI_at_20131013 into m_tank/RPI_at_20131013
> <118>19:45:12   5.90M   tank/RPI_at_20131013
> <118>19:45:13   36.4M   tank/RPI_at_20131013
> <118>19:45:15   38.4M   tank/RPI_at_20131013
> <118>19:45:16   41.3M   tank/RPI_at_20131013
> panic: solaris assert: dn->dn_datablkshift != 0, file:
> /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_tx.c,
> line: 638
> cpuid = 0
> KDB: stack backtrace:
> db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe00977711a0
> kdb_backtrace() at kdb_backtrace+0x39/frame 0xfffffe0097771250
> vpanic() at vpanic+0x126/frame 0xfffffe0097771290
> panic() at panic+0x43/frame 0xfffffe00977712f0
> assfail() at assfail+0x22/frame 0xfffffe0097771300
> dmu_tx_hold_free() at dmu_tx_hold_free+0x162/frame 0xfffffe00977713e0
> dmu_free_long_range() at dmu_free_long_range+0x244/frame 0xfffffe0097771450
> dmu_free_long_object() at dmu_free_long_object+0x1f/frame 0xfffffe0097771480
> dmu_recv_stream() at dmu_recv_stream+0x86e/frame 0xfffffe00977716b0
> zfs_ioc_recv() at zfs_ioc_recv+0x96c/frame 0xfffffe0097771920
> zfsdev_ioctl() at zfsdev_ioctl+0x54a/frame 0xfffffe00977719c0
> devfs_ioctl_f() at devfs_ioctl_f+0xf0/frame 0xfffffe0097771a20
> kern_ioctl() at kern_ioctl+0x2ca/frame 0xfffffe0097771a90
> sys_ioctl() at sys_ioctl+0x11f/frame 0xfffffe0097771ae0
> amd64_syscall() at amd64_syscall+0x265/frame 0xfffffe0097771bf0
> Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0097771bf0
> --- syscall (54, FreeBSD ELF64, sys_ioctl), rip = 0x8019f49ca, rsp =
> 0x7fffffff9438, rbp = 0x7fffffff94c0 ---
> KDB: enter: panic
> Uptime: 7m30s
> ...
> (kgdb) where
> #0  doadump (textdump=1) at pcpu.h:219
> #1  0xffffffff808b88b7 in kern_reboot (howto=260) at
> /usr/src/sys/kern/kern_shutdown.c:447
> #2  0xffffffff808b8dc5 in vpanic (fmt=<value optimized out>, ap=<value optimized
> out>) at /usr/src/sys/kern/kern_shutdown.c:754
> #3  0xffffffff808b8e13 in panic (fmt=<value optimized out>) at
> /usr/src/sys/kern/kern_shutdown.c:683
> #4  0xffffffff81dd1222 in assfail (a=<value optimized out>, f=<value optimized
> out>, l=<value optimized out>) at
> /usr/src/sys/modules/opensolaris/../../cddl/compat/opensolaris/kern/opensolaris_cmn_err.c:81
> 
> #5  0xffffffff81c847c2 in dmu_tx_hold_free (tx=0xfffff800118bda00, object=<value
> optimized out>, off=<value optimized out>, len=<value optimized out>)
>     at
> /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_tx.c:638
> 
> #6  0xffffffff81c78124 in dmu_free_long_range (os=0xfffff8000580f000,
> object=<value optimized out>, offset=0, length=18446744073709551615)
>     at
> /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu.c:654
> #7  0xffffffff81c781df in dmu_free_long_object (os=0xfffff8000580f000,
> object=66055) at
> /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu.c:700
> #8  0xffffffff81c7c39e in dmu_recv_stream (drc=0xfffffe0097771728, fp=<value
> optimized out>, voffp=0xfffffe0097771718, cleanup_fd=8, action_handlep=<value
> optimized out>)
>     at
> /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_send.c:1289
> 
> #9  0xffffffff81d0a1fc in zfs_ioc_recv (zc=0xfffffe0001830000) at
> /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_ioctl.c:4102
> 
> #10 0xffffffff81d054ea in zfsdev_ioctl (dev=<value optimized out>, zcmd=<value
> optimized out>, arg=<value optimized out>, flag=<value optimized out>, td=<value
> optimized out>)
>     at
> /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_ioctl.c:5960
> 
> #11 0xffffffff807b0d40 in devfs_ioctl_f (fp=0xfffff80005304dc0, com=3222821403,
> data=0xfffff800027abc40, cred=<value optimized out>, td=0xfffff8000524f000) at
> /usr/src/sys/fs/devfs/devfs_vnops.c:757
> #12 0xffffffff8090ffea in kern_ioctl (td=0xfffff8000524f000, fd=<value optimized
> out>, com=0) at file.h:319
> #13 0xffffffff8090fccf in sys_ioctl (td=0xfffff8000524f000,
> uap=0xfffffe0097771b80) at /usr/src/sys/kern/sys_generic.c:698
> #14 0xffffffff80cb2e05 in amd64_syscall (td=0xfffff8000524f000, traced=0) at
> subr_syscall.c:134
> #15 0xffffffff80c979fb in Xfast_syscall () at
> /usr/src/sys/amd64/amd64/exception.S:391
> #16 0x00000008019f49ca in ?? ()
> Previous frame inner to this frame (corrupt stack?)
> Current language:  auto; currently minimal
> (kgdb)
> 
> ...keith


-- 
Andriy Gapon