Re: ZFS panic with r255937

On Wed, 2 Oct 2013, Andriy Gapon wrote:

> on 30/09/2013 02:11 kwhite_at_site.uottawa.ca said the following:
>> Sorry, debugging this is *way* beyond me.  Any hints, patches to try?
>
> Please share the stack trace.
>
> -- 
> Andriy Gapon

There's now a pr for this panic: kern/182570

Here's the stack trace:

root_at_freebsd10:/usr/src # kgdb /boot/kernel/kernel /var/crash/vmcore.last
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd"...

Unread portion of the kernel message buffer:
panic: solaris assert: dn->dn_maxblkid == 0 && (BP_IS_HOLE(&dn->dn_phys->dn_blkptr[0]) || dnode_block_freed(dn, 0)), file: /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dnode.c, line: 598
cpuid = 1
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe00992b3280
kdb_backtrace() at kdb_backtrace+0x39/frame 0xfffffe00992b3330
vpanic() at vpanic+0x126/frame 0xfffffe00992b3370
panic() at panic+0x43/frame 0xfffffe00992b33d0
assfail() at assfail+0x22/frame 0xfffffe00992b33e0
dnode_reallocate() at dnode_reallocate+0x225/frame 0xfffffe00992b3430
dmu_object_reclaim() at dmu_object_reclaim+0x123/frame 0xfffffe00992b3480
dmu_recv_stream() at dmu_recv_stream+0xd79/frame 0xfffffe00992b36b0
zfs_ioc_recv() at zfs_ioc_recv+0x96c/frame 0xfffffe00992b3920
zfsdev_ioctl() at zfsdev_ioctl+0x54a/frame 0xfffffe00992b39c0
devfs_ioctl_f() at devfs_ioctl_f+0xf0/frame 0xfffffe00992b3a20
kern_ioctl() at kern_ioctl+0x2ca/frame 0xfffffe00992b3a90
sys_ioctl() at sys_ioctl+0x11f/frame 0xfffffe00992b3ae0
amd64_syscall() at amd64_syscall+0x265/frame 0xfffffe00992b3bf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe00992b3bf0
--- syscall (54, FreeBSD ELF64, sys_ioctl), rip = 0x8019dc9ca, rsp = 0x7fffffff5ad8, rbp = 0x7fffffff5b60 ---
KDB: enter: panic
Uptime: 37m10s
Dumping 874 out of 2023 MB:

...

(kgdb) where
#0  doadump (textdump=1) at pcpu.h:218
#1  0xffffffff808b7217 in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:447
#2  0xffffffff808b7725 in vpanic (fmt=<value optimized out>, ap=<value optimized out>) at /usr/src/sys/kern/kern_shutdown.c:754
#3  0xffffffff808b7773 in panic (fmt=<value optimized out>) at /usr/src/sys/kern/kern_shutdown.c:683
#4  0xffffffff81e52222 in assfail (a=<value optimized out>, f=<value optimized out>, l=<value optimized out>) at /usr/src/sys/modules/opensolaris/../../cddl/compat/opensolaris/kern/opensolaris_cmn_err.c:81
#5  0xffffffff81d09735 in dnode_reallocate (dn=0xfffff8006dde3000, ot=DMU_OT_PLAIN_FILE_CONTENTS, blocksize=1024, bonustype=DMU_OT_SA, bonuslen=168, tx=0xfffff8006d7a2600)
     at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dnode.c:596
#6  0xffffffff81cff463 in dmu_object_reclaim (os=<value optimized out>, object=<value optimized out>, ot=DMU_OT_PLAIN_FILE_CONTENTS, blocksize=1024, bonustype=DMU_OT_SA, bonuslen=168)
     at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_object.c:155
#7  0xffffffff81cfd849 in dmu_recv_stream (drc=0xfffffe00992b3728, fp=<value optimized out>, voffp=0xfffffe00992b3718, cleanup_fd=8, action_handlep=<value optimized out>)
     at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_send.c:1231
#8  0xffffffff81d8b1fc in zfs_ioc_recv (zc=0xfffffe00372e1000) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_ioctl.c:4102
#9  0xffffffff81d864ea in zfsdev_ioctl (dev=<value optimized out>, zcmd=<value optimized out>, arg=<value optimized out>, flag=<value optimized out>, td=<value optimized out>)
     at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_ioctl.c:5960
#10 0xffffffff807af840 in devfs_ioctl_f (fp=0xfffff800052e1460, com=3222821403, data=0xfffff8006ff1faa0, cred=<value optimized out>, td=0xfffff8003f7ba920) at /usr/src/sys/fs/devfs/devfs_vnops.c:757
#11 0xffffffff8090e94a in kern_ioctl (td=0xfffff8003f7ba920, fd=<value optimized out>, com=0) at file.h:319
#12 0xffffffff8090e62f in sys_ioctl (td=0xfffff8003f7ba920, uap=0xfffffe00992b3b80) at /usr/src/sys/kern/sys_generic.c:698
#13 0xffffffff80caee35 in amd64_syscall (td=0xfffff8003f7ba920, traced=0) at subr_syscall.c:134
#14 0xffffffff80c961ab in Xfast_syscall () at /usr/src/sys/amd64/amd64/exception.S:391
#15 0x00000008019dc9ca in ?? ()
Previous frame inner to this frame (corrupt stack?)
Current language:  auto; currently minimal
(kgdb)


Here's a how to repeat:

Assuming pool "tank" and non-existent tank/nobj tank/xobj

=== cut here ===
#!/bin/sh

zfs create tank/nobj
zfs snapshot tank/nobj_at_0
__MAKECONF=/dev/null SRCCONF=/dev/null MAKEOBJDIRPREFIX=/tank/nobj make -j6 buildkernel
zfs snapshot tank/nobj_at_1
#find /tank/nobj -name '.h' -print | xargs rm
zfs snapshot tank/nobj_at_2
#rm -rf /tank/nobj/*
zfs snapshot tank/nobj_at_3
__MAKECONF=/dev/null SRCCONF=/dev/null MAKEOBJDIRPREFIX=/tank/nobj make -j6 buildkernel
zfs snapshot tank/nobj_at_4
#rm -rf /tank/nobj/*
zfs snapshot tank/nobj_at_5

zfs send -vR tank/nobj_at_5 | zfs recv -vF tank/xobj

=== cut here ===

The commented out lines are historical attempts.

...keith