Re: BETA3 crash (zfs related ?)

On Nov 24, 2007 10:28 AM, Alexandre Biancalana <biancalana_at_gmail.com> wrote:
> On Nov 24, 2007 9:15 AM, Remko Lodder <remko_at_freebsd.org> wrote:
> >
> > Can you look at the developers handbook and follow the ways to get
> > a proper backtrace which we can investigate? I see from the logfile
> > that you send along, that a dump had been made, so the information
> > might be there :-)
> >
> > http://www.freebsd.org/doc/en/books/developers-handbook/kerneldebug.html
>
> My kernel is already compiled with all debug stuff. I forget to
> mention that I tried to debug but I can't look at this:
>
> Manny:/var/crash # kgdb /boot/kernel/kernel vmcore.6
> [GDB will not be able to debug user-mode threads:
> /usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"]
> GNU gdb 6.1.1 [FreeBSD]
> Copyright 2004 Free Software Foundation, Inc.
> GDB is free software, covered by the GNU General Public License, and you are
> welcome to change it and/or distribute copies of it under certain conditions.
> Type "show copying" to see the conditions.
> There is absolutely no warranty for GDB.  Type "show warranty" for details.
> This GDB was configured as "amd64-marcel-freebsd".
> (no debugging symbols found)...Attempt to extract a component of a
> value that is not a structure pointer.
> (kgdb) bt
> #0  0xffffffff8032668a in doadump ()
> #1  0xffffffffd8272450 in ?? ()
> #2  0xffffffff80326ba5 in boot ()
> Previous frame inner to this frame (corrupt stack?)
> (kgdb)
>
>
> Can you help with that ?
>
> Thank you !
>

Hi People !

Last night I get a similar crash, follow the messages from /var/log/messages


Nov 26 02:53:41 Manny kernel: uma_zalloc_arg: zone "256" with the
following non-sleepable locks held:
Nov 26 02:53:41 Manny kernel: exclusive sleep mutex struct mount mtx r
= 0 (0xffffff0003aba6a0) locked _at_
/usr/src/sys/modules/zfs/../../compat/opensolaris/kern/opensolaris_vfs.c:206
Nov 26 02:53:41 Manny kernel: KDB: stack backtrace:
Nov 26 02:53:41 Manny kernel: db_trace_self_wrapper() at
db_trace_self_wrapper+0x2a
Nov 26 02:53:41 Manny kernel: witness_warn() at witness_warn+0x248
Nov 26 02:53:41 Manny kernel: uma_zalloc_arg() at uma_zalloc_arg+0x33a
Nov 26 02:53:41 Manny kernel: malloc() at malloc+0x8a
Nov 26 02:53:41 Manny kernel: crget() at crget+0x1f
Nov 26 02:53:41 Manny kernel: crdup() at crdup+0x19
Nov 26 02:53:41 Manny kernel: domount() at domount+0x1fe
Nov 26 02:53:41 Manny kernel: zfsctl_snapdir_lookup() at
zfsctl_snapdir_lookup+0x307
Nov 26 02:53:41 Manny kernel: VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0x95
Nov 26 02:53:41 Manny kernel: lookup() at lookup+0x4cd
Nov 26 02:53:41 Manny kernel: namei() at namei+0x2cb
Nov 26 02:53:41 Manny kernel: kern_stat() at kern_stat+0x61
Nov 26 02:53:41 Manny kernel: stat() at stat+0x2a
Nov 26 02:53:41 Manny kernel: syscall() at syscall+0x1ce
Nov 26 02:53:41 Manny kernel: Xfast_syscall() at Xfast_syscall+0xab
Nov 26 02:53:41 Manny kernel: --- syscall (188, FreeBSD ELF64, stat),
rip = 0x800c6547c, rsp = 0x7fffffffe798, rbp = 0x2 ---
Nov 26 02:53:47 Manny kernel: uma_zalloc_arg: zone "256" with the
following non-sleepable locks held:
Nov 26 02:53:47 Manny kernel: exclusive sleep mutex struct mount mtx r
= 0 (0xffffff00032a96a0) locked _at_
/usr/src/sys/modules/zfs/../../compat/opensolaris/kern/opensolaris_vfs.c:206
Nov 26 02:53:47 Manny kernel: KDB: stack backtrace:
Nov 26 02:53:47 Manny kernel: db_trace_self_wrapper() at
db_trace_self_wrapper+0x2a
Nov 26 02:53:47 Manny kernel: witness_warn() at witness_warn+0x248
Nov 26 02:53:47 Manny kernel: uma_zalloc_arg() at uma_zalloc_arg+0x33a
Nov 26 02:53:47 Manny kernel: malloc() at malloc+0x8a
Nov 26 02:53:47 Manny kernel: crget() at crget+0x1f
Nov 26 02:53:47 Manny kernel: crdup() at crdup+0x19
Nov 26 02:53:47 Manny kernel: domount() at domount+0x1fe
Nov 26 02:53:47 Manny kernel: zfsctl_snapdir_lookup() at
zfsctl_snapdir_lookup+0x307
Nov 26 02:53:47 Manny kernel: VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0x95
Nov 26 02:53:47 Manny kernel: lookup() at lookup+0x4cd
Nov 26 02:53:47 Manny kernel: namei() at namei+0x2cb
Nov 26 02:53:47 Manny kernel: kern_stat() at kern_stat+0x61
Nov 26 02:53:47 Manny kernel: stat() at stat+0x2a
Nov 26 02:53:47 Manny kernel: syscall() at syscall+0x1ce
Nov 26 02:53:47 Manny kernel: Xfast_syscall() at Xfast_syscall+0xab
Nov 26 02:53:47 Manny kernel: --- syscall (188, FreeBSD ELF64, stat),
rip = 0x800c6547c, rsp = 0x7fffffffe798, rbp = 0x2 ---
Nov 26 02:53:48 Manny kernel: uma_zalloc_arg: zone "256" with the
following non-sleepable locks held:
Nov 26 02:53:48 Manny kernel: exclusive sleep mutex struct mount mtx r
= 0 (0xffffff00032a9378) locked _at_
/usr/src/sys/modules/zfs/../../compat/opensolaris/kern/opensolaris_vfs.c:206
Nov 26 02:53:48 Manny kernel: KDB: stack backtrace:
Nov 26 02:53:48 Manny kernel: db_trace_self_wrapper() at
db_trace_self_wrapper+0x2a
Nov 26 02:53:48 Manny kernel: witness_warn() at witness_warn+0x248
Nov 26 02:53:48 Manny kernel: uma_zalloc_arg() at uma_zalloc_arg+0x33a
Nov 26 02:53:48 Manny kernel: malloc() at malloc+0x8a
Nov 26 02:53:48 Manny kernel: crget() at crget+0x1f
Nov 26 02:53:48 Manny kernel:
Nov 26 02:53:48 Manny kernel: crdup() at
Nov 26 02:53:48 Manny kernel: crdup+0x19
Nov 26 02:53:48 Manny kernel: domount() at domount+0x1fe
Nov 26 02:53:48 Manny kernel: zfsctl_snapdir_lookup() at zfsctl_snapdir_lookup
Nov 26 02:53:48 Manny kernel: +0x307
Nov 26 02:53:48 Manny kernel:
Nov 26 02:53:48 Manny kernel: VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0x95
Nov 26 02:53:48 Manny kernel: lookup() at
Nov 26 02:53:48 Manny kernel: lookup+0x4cd
Nov 26 02:53:48 Manny kernel: namei() at namei+0x2cb
Nov 26 02:53:48 Manny kernel: kern_stat() at kern_stat+0x61
Nov 26 02:53:48 Manny kernel:
Nov 26 02:53:48 Manny kernel: stat() at stat+0x2a
Nov 26 02:53:48 Manny kernel: syscall() at
Nov 26 02:53:48 Manny kernel: syscall+0x1ce
Nov 26 02:53:48 Manny kernel: Xfast_sys
Nov 26 02:53:48 Manny kernel: call() at
Nov 26 02:53:48 Manny kernel: Xfast_syscall+0xab
Nov 26 02:53:48 Manny kernel: --- syscall (188, FreeBSD ELF64, stat),
rip = 0x800c6547c, rsp = 0x7fffffffe79
Nov 26 02:53:48 Manny kernel: 8
Nov 26 02:53:48 Manny kernel: , rbp = 0x2 ---
Nov 26 02:56:34 Manny kernel: uma_zalloc_arg: zone "256" with the
following non-sleepable locks held:
Nov 26 02:56:34 Manny kernel: exclusive sleep mutex struct mount mtx r
= 0 (0xffffff0003ab96a0) locked _at_
/usr/src/sys/modules/zfs/../../compat/opensolaris/kern/opensolaris_vfs.c:206
Nov 26 02:56:34 Manny kernel: KDB: stack backtrace:
Nov 26 02:56:34 Manny kernel: db_trace_self_wrapper() at
db_trace_self_wrapper+0x2a
Nov 26 02:56:34 Manny kernel: witness_warn() at witness_warn+0x248
Nov 26 02:56:34 Manny kernel: uma_zalloc_arg() at uma_zalloc_arg+0x33a
Nov 26 02:56:34 Manny kernel: malloc() at malloc+0x8a
Nov 26 02:56:34 Manny kernel: crget() at crget+0x1f
Nov 26 02:56:34 Manny kernel: crdup() at crdup+0x19
Nov 26 02:56:34 Manny kernel: domount() at domount+0x1fe
Nov 26 02:56:34 Manny kernel: zfsctl_snapdir_lookup() at
zfsctl_snapdir_lookup+0x307
Nov 26 02:56:34 Manny kernel: VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0x95
Nov 26 02:56:34 Manny kernel: lookup() at lookup+0x4cd
Nov 26 02:56:34 Manny kernel: namei() at namei+0x2cb
Nov 26 02:56:34 Manny kernel: kern_stat() at kern_stat+0x61
Nov 26 02:56:34 Manny kernel: stat() at stat+0x2a
Nov 26 02:56:34 Manny kernel: syscall() at syscall+0x1ce
Nov 26 02:56:34 Manny kernel: Xfast_syscall() at Xfast_syscall+0xab
Nov 26 02:56:34 Manny kernel: --- syscall (188, FreeBSD ELF64, stat),
rip = 0x800c6547c, rsp = 0x7fffffffe798, rbp = 0x2 ---
Nov 26 02:56:37 Manny kernel: uma_zalloc_arg: zone "256" with the
following non-sleepable locks held:
Nov 26 02:56:37 Manny kernel: exclusive sleep mutex struct mount mtx r
= 0 (0xffffff0003ab9378) locked _at_
/usr/src/sys/modules/zfs/../../compat/opensolaris/kern/opensolaris_vfs.c:206
Nov 26 02:56:37 Manny kernel: KDB: stack backtrace:
Nov 26 02:56:37 Manny kernel: db_trace_self_wrapper() at
db_trace_self_wrapper+0x2a
Nov 26 02:56:37 Manny kernel: witness_warn() at witness_warn+0x248
Nov 26 02:56:37 Manny kernel: uma_zalloc_arg() at uma_zalloc_arg+0x33a
Nov 26 02:56:37 Manny kernel: malloc() at malloc+0x8a
Nov 26 02:56:37 Manny kernel: crget() at crget+0x1f
Nov 26 02:56:37 Manny kernel: crdup() at crdup+0x19
Nov 26 02:56:37 Manny kernel: domount() at domount+0x1fe
Nov 26 02:56:37 Manny kernel: zfsctl_snapdir_lookup() at
Nov 26 02:56:37 Manny kernel: zfsctl_snapdir_lookup+0x307
Nov 26 02:56:37 Manny kernel: VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0x95
Nov 26 02:56:37 Manny kernel:
Nov 26 02:56:37 Manny kernel: lookup() at lookup+0x4cd
Nov 26 02:56:37 Manny kernel: namei() at
Nov 26 02:56:37 Manny kernel: namei+0x2cb
Nov 26 02:56:37 Manny kernel: k
Nov 26 02:56:37 Manny kernel: ern_stat
Nov 26 02:56:37 Manny kernel: () at
Nov 26 02:56:37 Manny kernel: kern_stat+0x61
Nov 26 02:56:37 Manny kernel: stat() at stat+0x2a
Nov 26 02:56:37 Manny kernel:
Nov 26 02:56:37 Manny kernel: syscall() at
Nov 26 02:56:37 Manny kernel: syscall+0x1ce
Nov 26 02:56:37 Manny kernel: Xfast_syscall() at
Nov 26 02:56:37 Manny kernel: Xfast_syscall+0xab
Nov 26 02:56:37 Manny kernel: --- syscall (188, FreeBSD ELF64, stat),
rip = 0x800c6547c, rsp = 0x7fffffffe798, rbp = 0x2 ---
Nov 26 02:56:37 Manny kernel: uma_zalloc_arg: zone "256" with the
following non-sleepable locks held:
Nov 26 02:56:37 Manny kernel: exclusive sleep mutex struct mount mtx r
= 0 (0xffffff0003ab9050) locked _at_
/usr/src/sys/modules/zfs/../../compat/opensolaris/kern/opensolaris_vfs.c:206
Nov 26 02:56:37 Manny kernel: KDB: stack backtrace:
Nov 26 02:56:37 Manny kernel: db_trace_self_wrapper() at
db_trace_self_wrapper+0x2a
Nov 26 02:56:37 Manny kernel: witness_warn() at witness_warn+0x248
Nov 26 02:56:37 Manny kernel: uma_zalloc_arg() at uma_zalloc_arg+0x33a
Nov 26 02:56:37 Manny kernel: malloc() at malloc+0x8a
Nov 26 02:56:37 Manny kernel: crget() at crget+0x1f
Nov 26 02:56:37 Manny kernel: crdup() at crdup+0x19
Nov 26 02:56:37 Manny kernel: domount() at domount+0x1fe
Nov 26 02:56:37 Manny kernel: zfsctl_snapdir_lookup() at
zfsctl_snapdir_lookup+0x307
Nov 26 02:56:37 Manny kernel: VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0x95
Nov 26 02:56:37 Manny kernel: lookup() at lookup+0x4cd
Nov 26 02:56:37 Manny kernel: namei() at namei+0x2cb
Nov 26 02:56:37 Manny kernel: kern_stat() at kern_stat+0x61
Nov 26 02:56:37 Manny kernel: stat() at stat+0x2a
Nov 26 02:56:37 Manny kernel: syscall() at syscall+0x1ce
Nov 26 02:56:37 Manny kernel: Xfast_syscall() at Xfast_syscall+0xab
Nov 26 02:56:37 Manny kernel: --- syscall (188, FreeBSD ELF64, stat),
rip = 0x800c6547c, rsp = 0x7fffffffe798, rbp = 0x2 ---
Nov 26 02:56:37 Manny kernel: uma_zalloc_arg: zone "256" with the
following non-sleepable locks held:
Nov 26 02:56:37 Manny kernel: exclusive sleep mutex struct mount mtx r
= 0 (0xffffff0003944050) locked _at_
/usr/src/sys/modules/zfs/../../compat/opensolaris/kern/opensolaris_vfs.c:206
Nov 26 02:56:37 Manny kernel: KDB: stack backtrace:
Nov 26 02:56:37 Manny kernel: db_trace_self_wrapper() at
db_trace_self_wrapper+0x2a
Nov 26 02:56:37 Manny kernel: witness_warn() at witness_warn+0x248
Nov 26 02:56:37 Manny kernel: uma_zalloc_arg() at uma_zalloc_arg+0x33a
Nov 26 02:56:37 Manny kernel: malloc() at malloc+0x8a
Nov 26 02:56:37 Manny kernel: crget() at crget+0x1f
Nov 26 02:56:37 Manny kernel: crdup() at crdup+0x19
Nov 26 02:56:37 Manny kernel: domount() at domount+0x1fe
Nov 26 02:56:37 Manny kernel: zfsctl_snapdir_lookup() at
zfsctl_snapdir_lookup+0x307
Nov 26 02:56:37 Manny kernel: VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0x95
Nov 26 02:56:37 Manny kernel: lookup() at
Nov 26 02:56:37 Manny kernel: lookup+0x4cd
Nov 26 02:56:37 Manny kernel: namei() at namei+0x2cb
Nov 26 02:56:37 Manny kernel:
Nov 26 02:56:37 Manny kernel: kern_stat() at kern_stat+0x61
Nov 26 02:56:37 Manny kernel: stat() at stat+0x2a
Nov 26 02:56:37 Manny kernel:
Nov 26 02:56:37 Manny kernel: syscall() at
Nov 26 02:56:37 Manny kernel: syscall+0x1ce
Nov 26 02:56:37 Manny kernel: Xfast_syscall() at Xfast_syscall+0xab
Nov 26 02:56:37 Manny kernel:
Nov 26 02:56:37 Manny kernel: --- syscall (1
Nov 26 02:56:37 Manny kernel: 88
Nov 26 02:56:37 Manny kernel: , FreeBSD ELF64, stat), rip =
0x800c6547c, rsp = 0x7fffffffe798, rbp = 0x2 ---
Nov 26 02:56:44 Manny kernel: uma_zalloc_arg: zone "256" with the
following non-sleepable locks held:
Nov 26 02:56:44 Manny kernel: exclusive sleep mutex struct mount mtx r
= 0 (0xffffff00039b06a0) locked _at_
/usr/src/sys/modules/zfs/../../compat/opensolaris/kern/opensolaris_vfs.c:206
Nov 26 02:56:44 Manny kernel: KDB: stack backtrace:
Nov 26 02:56:44 Manny kernel: db_trace_self_wrapper() at
db_trace_self_wrapper+0x2a
Nov 26 02:56:44 Manny kernel: witness_warn() at witness_warn+0x248
Nov 26 02:56:44 Manny kernel: uma_zalloc_arg() at uma_zalloc_arg+0x33a
Nov 26 02:56:44 Manny kernel: malloc() at malloc+0x8a
Nov 26 02:56:44 Manny kernel: crget() at crget+0x1f
Nov 26 02:56:44 Manny kernel: crdup() at crdup+0x19
Nov 26 02:56:44 Manny kernel: domount() at domount+0x1fe
Nov 26 02:56:44 Manny kernel: zfsctl_snapdir_lookup() at
zfsctl_snapdir_lookup+0x307
Nov 26 02:56:44 Manny kernel: VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0x95
Nov 26 02:56:44 Manny kernel: lookup() at lookup+0x4cd
Nov 26 02:56:44 Manny kernel: namei() at namei+0x2cb
Nov 26 02:56:44 Manny kernel: kern_stat() at kern_stat+0x61
Nov 26 02:56:44 Manny kernel: stat() at stat+0x2a
Nov 26 02:56:44 Manny kernel: syscall() at syscall+0x1ce
Nov 26 02:56:44 Manny kernel: Xfast_syscall() at Xfast_syscall+0xab
Nov 26 02:56:44 Manny kernel: --- syscall (188, FreeBSD ELF64, stat),
rip = 0x800c6547c, rsp = 0x7fffffffe798
Nov 26 02:56:44 Manny kernel: , rbp = 0x2 ---
Nov 26 03:10:55 Manny syslogd: kernel boot file is /boot/kernel/kernel
Nov 26 03:10:55 Manny kernel: panic: vm_fault: fault on nofault entry,
addr: ffffffffd8547000
Nov 26 03:10:55 Manny kernel: cpuid = 1
Nov 26 03:10:55 Manny kernel: KDB: stack backtrace:
Nov 26 03:10:55 Manny kernel: db_trace_self_wrapper() at
db_trace_self_wrapper+0x2a
Nov 26 03:10:55 Manny kernel: panic() at panic+0x17a
Nov 26 03:10:55 Manny kernel: vm_fault() at vm_fault+0x14c1
Nov 26 03:10:55 Manny kernel: trap_pfault() at trap_pfault+0x218
Nov 26 03:10:55 Manny kernel: trap() at trap+0x30c
Nov 26 03:10:55 Manny kernel: calltrap() at calltrap+0x8
Nov 26 03:10:55 Manny kernel: --- trap 0xc, rip = 0xffffffffd8198e54,
rsp = 0xffffffffd8168960, rbp = 0xffffffffd81689a0 ---
Nov 26 03:10:55 Manny kernel: dsl_dir_set_reservation_check() at
dsl_dir_set_reservation_check+0x24
Nov 26 03:10:55 Manny kernel: dsl_sync_task_group_sync() at
dsl_sync_task_group_sync+0x96
Nov 26 03:10:55 Manny kernel: dsl_pool_sync() at dsl_pool_sync+0xc3
Nov 26 03:10:55 Manny kernel: spa_sync() at spa_sync+0x390
Nov 26 03:10:55 Manny kernel: txg_sync_thread() at txg_sync_thread+0x12f
Nov 26 03:10:55 Manny kernel: fork_exit() at fork_exit+0x12a
Nov 26 03:10:55 Manny kernel: fork_trampoline() at fork_trampoline+0xe
Nov 26 03:10:55 Manny kernel: --- trap 0, rip = 0, rsp =
0xffffffffd8168d30, rbp = 0 ---
Nov 26 03:10:55 Manny kernel: Uptime: 12h55m27s
Nov 26 03:10:55 Manny kernel: Physical memory: 2036 MB
Nov 26 03:10:55 Manny kernel: Dumping 1178 MB: 1163 1147 1131 1115
1099 1083 1067 1051 1035 1019 1003 987 971 955 939 923 907 891 875 859
843 827 811 795 779 763 747 731 715 699 683 667 651 635 619 603 587
571 555 539 523 507 491 475 459 443 427 411 395 379 363 347 331 315
299 283 267 251 235 219 203 187 171 155 139 123 107 91 75 59 43 27 11
Nov 26 03:10:55 Manny kernel: Dump complete
Nov 26 03:10:55 Manny kernel: Automatic reboot in 15 seconds - press a
key on the console to abort
Nov 26 03:10:55 Manny kernel: Rebooting...
Nov 26 03:10:55 Manny kernel: cpu_reset: Stopping other CPUs