Re: zfs: Fatal trap 12: page fault while in kernel mode

On Jul 30, 2009, at 14:11, Andriy Gapon wrote:

> on 29/07/2009 21:04 Thomas Backman said the following:
>> Thanks for your work :)
>> However, bad news: it didn't help. It *might* have gotten us further,
>> though, because the DDB backtrace now looks like this:
>>
>> _sx_xlock_hard()
>> _sx_xlock()
>> zfs_znode_free()
>> zfs_freebsd_inactive()
>> VOP_INACTIVE_APV()
>> vinactive()
>> vput()
>> dounmount()
>> unmount()
>> syscall()
>> XFast_syscall()
>>
>
> Oh my bad. I missed the fact that recycle would do zfs_znode_free,  
> so it seems
> like zfs_znode_free was called twice on the same znode.
> Could you please try replacing
> 	zfs_znode_free(zp);
> with
> 	vrecycle(vp, curthread);
> in the same block (instead of adding the latter before the former).
> Sorry, if this looks like shooting in the dark - because this is  
> what it is. I am
> not familiar with the code and it's hard to follow all possibilities  
> without good
> understanding.

New panic. :( Damnit!

I think I'm using svn + http://people.freebsd.org/~pjd/patches/zfs_vnops.c.2.patch 
  + your change, now...

Unread portion of the kernel message buffer:
GEOM_GATE: Device ggate1482 destroyed.
panic: solaris assert: zp != ((void *)0), file: /usr/src/sys/modules/ 
zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c,  
line: 4359
cpuid = 0
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
panic() at panic+0x182
zfs_freebsd_reclaim() at zfs_freebsd_reclaim+0x244
VOP_RECLAIM_APV() at VOP_RECLAIM_APV+0x4a
vgonel() at vgonel+0x12e
vrecycle() at vrecycle+0x7d
zfs_freebsd_inactive() at zfs_freebsd_inactive+0x1a
VOP_INACTIVE_APV() at VOP_INACTIVE_APV+0x4a
vinactive() at vinactive+0x6a
vput() at vput+0x1c6
dounmount() at dounmount+0x4af
unmount() at unmount+0x3c8
syscall() at syscall+0x28f
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (22, FreeBSD ELF64, unmount), rip = 0x80104e9ec, rsp =  
0x7fffffffaa98, rbp = 0x801223300 ---
KDB: enter: panic

0xffffff00452971d8: tag zfs, type VDIR
     usecount 0, writecount 0, refcount 1 mountedhere 0
     flags (VI_DOOMED|VI_DOINGINACT)    lock type zfs: EXCL by thread  
0xffffff0019ff6000 (pid 1425)
panic: from debugger
...

#11 0xffffffff8033a9cb in panic (fmt=Variable "fmt" is not available.
)
     at /usr/src/sys/kern/kern_shutdown.c:558
#12 0xffffffff80b11124 in zfs_freebsd_reclaim () from /boot/kernel/ 
zfs.ko
#13 0xffffffff805c5c2a in VOP_RECLAIM_APV (vop=0x0,  
a=0xffffff803eaf8930)
     at vnode_if.c:1926
#14 0xffffffff803c839e in vgonel (vp=0xffffff00452971d8) at vnode_if.h: 
830
#15 0xffffffff803ca7ad in vrecycle (vp=0xffffff00452971d8, td=Variable  
"td" is not available.
)
     at /usr/src/sys/kern/vfs_subr.c:2504
#16 0xffffffff80b10aaa in zfs_freebsd_inactive () from /boot/kernel/ 
zfs.ko
#17 0xffffffff805c5b5a in VOP_INACTIVE_APV (vop=0xffffffff80b882a0,
     a=0xffffff803eaf89f0) at vnode_if.c:1863
#18 0xffffffff803c6aaa in vinactive (vp=0xffffff00452971d8,
     td=0xffffff0019ff6000) at vnode_if.h:807
#19 0xffffffff803cbf26 in vput (vp=0xffffff00452971d8)
     at /usr/src/sys/kern/vfs_subr.c:2257
#20 0xffffffff803c57ef in dounmount (mp=0xffffff0001d058d0, flags=0,  
td=Variable "td" is not available.
)
     at /usr/src/sys/kern/vfs_mount.c:1333
#21 0xffffffff803c5df8 in unmount (td=0xffffff0019ff6000,
     uap=0xffffff803eaf8bf0) at /usr/src/sys/kern/vfs_mount.c:1174
#22 0xffffffff805980bf in syscall (frame=0xffffff803eaf8c80)
     at /usr/src/sys/amd64/amd64/trap.c:984
#23 0xffffffff8057e2c1 in Xfast_syscall ()
     at /usr/src/sys/amd64/amd64/exception.S:373
#24 0x000000080104e9ec in ?? ()

FWIW:
Line 4359 (panic line):
zfs_freebsd_reclaim(ap) ... {
   vnode_t *vp = ap->a_vp;
   znode_t *zp = VTOZ(vp);

     ASSERT(ap != NULL); // added by me
     ASSERT(vp != NULL); // added by me
 >>>    ASSERT(zp != NULL); // line 4359

---------------

zfs_inactive(vnode_t *vp, cred_t *cr, caller_context_t *ct)
{
     znode_t *zp = VTOZ(vp);
     zfsvfs_t *zfsvfs = zp->z_zfsvfs;
     int error;

     rw_enter(&zfsvfs->z_teardown_inactive_lock, RW_READER);
     if (zp->z_dbuf == NULL) {
         /*
          * The fs has been unmounted, or we did a
          * suspend/resume and this file no longer exists.
          */
         VI_LOCK(vp);
         vp->v_count = 0; /* count arrives as 1 */
         vp->v_data = NULL;
         VI_UNLOCK(vp);
         rw_exit(&zfsvfs->z_teardown_inactive_lock);
         ZTOV(zp) = NULL;
         vrecycle(vp, curthread);
//      zfs_znode_free(zp);
         return;
     }

Regards,
Thomas

PS.
... and thanks again for working to solve this. :)