Re: ZFS panic with r255937

on 02/10/2013 20:59 Keith White said the following:
> On Wed, 2 Oct 2013, Andriy Gapon wrote:
> 
>> on 30/09/2013 02:11 kwhite_at_site.uottawa.ca said the following:
>>> Sorry, debugging this is *way* beyond me.  Any hints, patches to try?
>>
>> Please share the stack trace.
>>
>> -- 
>> Andriy Gapon
> 
> There's now a pr for this panic: kern/182570
> 
> Here's the stack trace:
> 
> root_at_freebsd10:/usr/src # kgdb /boot/kernel/kernel /var/crash/vmcore.last
> GNU gdb 6.1.1 [FreeBSD]
> Copyright 2004 Free Software Foundation, Inc.
> GDB is free software, covered by the GNU General Public License, and you are
> welcome to change it and/or distribute copies of it under certain conditions.
> Type "show copying" to see the conditions.
> There is absolutely no warranty for GDB.  Type "show warranty" for details.
> This GDB was configured as "amd64-marcel-freebsd"...
> 
> Unread portion of the kernel message buffer:
> panic: solaris assert: dn->dn_maxblkid == 0 &&
> (BP_IS_HOLE(&dn->dn_phys->dn_blkptr[0]) || dnode_block_freed(dn, 0)), file:
> /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dnode.c,
> line: 598
> cpuid = 1
> KDB: stack backtrace:
> db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe00992b3280
> kdb_backtrace() at kdb_backtrace+0x39/frame 0xfffffe00992b3330
> vpanic() at vpanic+0x126/frame 0xfffffe00992b3370
> panic() at panic+0x43/frame 0xfffffe00992b33d0
> assfail() at assfail+0x22/frame 0xfffffe00992b33e0
> dnode_reallocate() at dnode_reallocate+0x225/frame 0xfffffe00992b3430
> dmu_object_reclaim() at dmu_object_reclaim+0x123/frame 0xfffffe00992b3480
> dmu_recv_stream() at dmu_recv_stream+0xd79/frame 0xfffffe00992b36b0
> zfs_ioc_recv() at zfs_ioc_recv+0x96c/frame 0xfffffe00992b3920
> zfsdev_ioctl() at zfsdev_ioctl+0x54a/frame 0xfffffe00992b39c0
> devfs_ioctl_f() at devfs_ioctl_f+0xf0/frame 0xfffffe00992b3a20
> kern_ioctl() at kern_ioctl+0x2ca/frame 0xfffffe00992b3a90
> sys_ioctl() at sys_ioctl+0x11f/frame 0xfffffe00992b3ae0
> amd64_syscall() at amd64_syscall+0x265/frame 0xfffffe00992b3bf0
> Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe00992b3bf0


Thank you very much.
To me this looks very similar to a problem discovered and fixed in illumos some
time ago.  Please check if the following change improves the situation for you.

https://github.com/avg-I/freebsd/commit/a7e7dece215bc5d00077e9c7f4db34d9e5c30659

Raw:
https://github.com/avg-I/freebsd/commit/a7e7dece215bc5d00077e9c7f4db34d9e5c30659.patch

> --- syscall (54, FreeBSD ELF64, sys_ioctl), rip = 0x8019dc9ca, rsp =
> 0x7fffffff5ad8, rbp = 0x7fffffff5b60 ---
> KDB: enter: panic
> Uptime: 37m10s
> Dumping 874 out of 2023 MB:
> 
> ...
> 
> (kgdb) where
> #0  doadump (textdump=1) at pcpu.h:218
> #1  0xffffffff808b7217 in kern_reboot (howto=260) at
> /usr/src/sys/kern/kern_shutdown.c:447
> #2  0xffffffff808b7725 in vpanic (fmt=<value optimized out>, ap=<value optimized
> out>) at /usr/src/sys/kern/kern_shutdown.c:754
> #3  0xffffffff808b7773 in panic (fmt=<value optimized out>) at
> /usr/src/sys/kern/kern_shutdown.c:683
> #4  0xffffffff81e52222 in assfail (a=<value optimized out>, f=<value optimized
> out>, l=<value optimized out>) at
> /usr/src/sys/modules/opensolaris/../../cddl/compat/opensolaris/kern/opensolaris_cmn_err.c:81
> 
> #5  0xffffffff81d09735 in dnode_reallocate (dn=0xfffff8006dde3000,
> ot=DMU_OT_PLAIN_FILE_CONTENTS, blocksize=1024, bonustype=DMU_OT_SA,
> bonuslen=168, tx=0xfffff8006d7a2600)
>     at
> /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dnode.c:596
> 
> #6  0xffffffff81cff463 in dmu_object_reclaim (os=<value optimized out>,
> object=<value optimized out>, ot=DMU_OT_PLAIN_FILE_CONTENTS, blocksize=1024,
> bonustype=DMU_OT_SA, bonuslen=168)
>     at
> /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_object.c:155
> 
> #7  0xffffffff81cfd849 in dmu_recv_stream (drc=0xfffffe00992b3728, fp=<value
> optimized out>, voffp=0xfffffe00992b3718, cleanup_fd=8, action_handlep=<value
> optimized out>)
>     at
> /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_send.c:1231
> 
> #8  0xffffffff81d8b1fc in zfs_ioc_recv (zc=0xfffffe00372e1000) at
> /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_ioctl.c:4102
> 
> #9  0xffffffff81d864ea in zfsdev_ioctl (dev=<value optimized out>, zcmd=<value
> optimized out>, arg=<value optimized out>, flag=<value optimized out>, td=<value
> optimized out>)
>     at
> /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_ioctl.c:5960
> 
> #10 0xffffffff807af840 in devfs_ioctl_f (fp=0xfffff800052e1460, com=3222821403,
> data=0xfffff8006ff1faa0, cred=<value optimized out>, td=0xfffff8003f7ba920) at
> /usr/src/sys/fs/devfs/devfs_vnops.c:757
> #11 0xffffffff8090e94a in kern_ioctl (td=0xfffff8003f7ba920, fd=<value optimized
> out>, com=0) at file.h:319
> #12 0xffffffff8090e62f in sys_ioctl (td=0xfffff8003f7ba920,
> uap=0xfffffe00992b3b80) at /usr/src/sys/kern/sys_generic.c:698
> #13 0xffffffff80caee35 in amd64_syscall (td=0xfffff8003f7ba920, traced=0) at
> subr_syscall.c:134
> #14 0xffffffff80c961ab in Xfast_syscall () at
> /usr/src/sys/amd64/amd64/exception.S:391
> #15 0x00000008019dc9ca in ?? ()
> Previous frame inner to this frame (corrupt stack?)
> Current language:  auto; currently minimal
> (kgdb)
> 
> 
> Here's a how to repeat:
> 
> Assuming pool "tank" and non-existent tank/nobj tank/xobj
> 
> === cut here ===
> #!/bin/sh
> 
> zfs create tank/nobj
> zfs snapshot tank/nobj_at_0
> __MAKECONF=/dev/null SRCCONF=/dev/null MAKEOBJDIRPREFIX=/tank/nobj make -j6
> buildkernel
> zfs snapshot tank/nobj_at_1
> #find /tank/nobj -name '.h' -print | xargs rm
> zfs snapshot tank/nobj_at_2
> #rm -rf /tank/nobj/*
> zfs snapshot tank/nobj_at_3
> __MAKECONF=/dev/null SRCCONF=/dev/null MAKEOBJDIRPREFIX=/tank/nobj make -j6
> buildkernel
> zfs snapshot tank/nobj_at_4
> #rm -rf /tank/nobj/*
> zfs snapshot tank/nobj_at_5
> 
> zfs send -vR tank/nobj_at_5 | zfs recv -vF tank/xobj
> 
> === cut here ===
> 
> The commented out lines are historical attempts.
> 
> ...keith


-- 
Andriy Gapon