Re: zfs: Fatal trap 12: page fault while in kernel mode

On Jul 29, 2009, at 15:03, Andriy Gapon wrote:

> on 29/07/2009 15:45 Thomas Backman said the following:
>> On Jul 29, 2009, at 13:21, Andriy Gapon wrote:
>>
>>> on 29/07/2009 13:32 Thomas Backman said the following:
>>>> OFF TOPIC:
>>>> Due to similarities in the backtrace between this and a panic  
>>>> I've been
>>>> seeing on exporting after zfs recv (see
>>>> http://lists.freebsd.org/pipermail/freebsd-current/2009-July/009105.html 
>>>>  and
>>>>
>>>> also
>>>> http://lists.freebsd.org/pipermail/freebsd-current/2009-July/009174.html 
>>>>  for
>>>>
> [snip]
>> The DDB output from one panic does involve zfs_znode_dmu_fini and
>> dmu_buf_update_user:
>> _sx_xlock()
>> dmu_buf_update_user()+0x47
>> zfs_znode_dmu_fini()
>> zfs_freebsd_reclaim()
>> VOP_RECLAIM_APV()
>> vgonel()
>> vflush()
>> zfs_umount()
>> dounmount()
>> unmount()
>> syscall()
>> Xfast_syscall()
>> (Sorry if the formatting got screwed up above.)
>
> Hmm, then you experienced two different kinds of panics.
> To quote the link you posted earlier:
> [1] http://lists.freebsd.org/pipermail/freebsd-current/2009-July/009174.html
> ...
> #10 0xffffffff8086e007 in calltrap () at /usr/src/sys/amd64/amd64/ 
> exception.S:223
> #11 0xffffffff805a4989 in _sx_xlock_hard (sx=0xffffff0043557d50,
> tid=18446742975830720512, opts=Variable "opts" is not available.) at
> /usr/src/sys/kern/kern_sx.c:575
> #12 0xffffffff805a52fe in _sx_xlock (sx=Variable "sx" is not  
> available.) at sx.h:155
> #13 0xffffffff80fe2995 in zfs_freebsd_reclaim () from /boot/kernel/ 
> zfs.ko
> ...
>
> So now that you said that the patch didn't fix the problem for you,  
> could you
> please clarify what panic you do see after applying it?
> [...]
>
> Ok, then if you get a crash dump, you are able use kgdb and it will  
> be able to
> produce line numbers and will allow to examine variables.
>
> P.S. sorry if I miss context of your previous reports.
>
> -- 
> Andriy Gapon
Hmm, you are indeed right, it's not the same panic. The backtrace I  
got just now with INVARIANTS is the one you quoted above.
I still get the "_sx_xlock (sx=Variable "sx" is not available.)" and  
"_sx_xlock_hard (sx=0xffffff00090d5018, ..., opts=Variable "opts" is  
not available.)" though.
Am I missing some option (I've got GENERIC, minus WITNESS plus DTRACE,  
now that INVARIANTS is back in place), or does this "just happen"?

Here's the "full" backtrace (minus the panic(), trap() etc.):

#10 0xffffffff8057dfe7 in calltrap ()
     at /usr/src/sys/amd64/amd64/exception.S:224
#11 0xffffffff80342b99 in _sx_xlock_hard (sx=0xffffff00090d5018,
     tid=18446742974952890368, opts=Variable "opts" is not available.
) at /usr/src/sys/kern/kern_sx.c:575
#12 0xffffffff8034350e in _sx_xlock (sx=Variable "sx" is not available.
) at sx.h:155
#13 0xffffffff80af7596 in zfs_freebsd_reclaim () from /boot/kernel/ 
zfs.ko
#14 0xffffffff805c5c2a in VOP_RECLAIM_APV (vop=0xffffff00090d5018,
     a=0xffffff00090d5000) at vnode_if.c:1926
#15 0xffffffff803c839e in vgonel (vp=0xffffff0009252588) at vnode_if.h: 
830
#16 0xffffffff803cc958 in vflush (mp=0xffffff0002cd7bc0, rootrefs=0,  
flags=0,
     td=0xffffff002cffe000) at /usr/src/sys/kern/vfs_subr.c:2449
#17 0xffffffff80af2038 in zfs_umount () from /boot/kernel/zfs.ko
#18 0xffffffff803c55ca in dounmount (mp=0xffffff0002cd7bc0,  
flags=47020992,
     td=Variable "td" is not available.
) at /usr/src/sys/kern/vfs_mount.c:1289
#19 0xffffffff803c5df8 in unmount (td=0xffffff002cffe000,
     uap=0xffffff803e98bbf0) at /usr/src/sys/kern/vfs_mount.c:1174
#20 0xffffffff805980bf in syscall (frame=0xffffff803e98bc80)
     at /usr/src/sys/amd64/amd64/trap.c:984
#21 0xffffffff8057e2c1 in Xfast_syscall ()
     at /usr/src/sys/amd64/amd64/exception.S:373
#22 0x000000080104e9ec in ?? ()
Previous frame inner to this frame (corrupt stack?)

This happens on "zpool export" on the receiving pool (never on the  
sending pool) when running the script in the posts above. (Which, I  
realize, few people will have run.)
I also get another panic when manually doing zfs unmount on the root  
FS on the pool, rather than exporting it: http://lists.freebsd.org/pipermail/freebsd-current/2009-July/009209.html

Now we're drifting way off topic, though. Sorry.

Regards,
Thomas