Re: GPF on Current kernel

From: K. Macy <kmacy_at_freebsd.org>
Date: Mon, 6 Jul 2015 15:43:02 -0700
That's really disturbing, but could be an artifact of limitations in debug
info fidelity. If curthread is corrupt I would expect things to blow up
almost instantly.

Try enabling memguard for the zone or mtype for the object that has the
corrupt mutex field.

-K
 On Jul 6, 2015 5:48 AM, "Larry Rosenman" <ler_at_lerctr.org> wrote:

> actually, the following are interesting further down:
> #13 0xffffffff808af473 in sys_write (td=0xdeadc0dedeadc116,
>     uap=<value optimized out>) at /usr/src/sys/kern/sys_generic.c:398
> #14 0xffffffff80b57a6d in amd64_syscall (td=0xfffff80736fda4c0, traced=0)
>     at subr_syscall.c:133
>
> the td value.
>
> What would you suggest?  This has become intermittent :(
>
>
> On 2015-07-06 00:42, K. Macy wrote:
>
>> The value of the mutex in the stack trace indicates that it's a use after
>> free. There are various memory debugging options (memguard, redzone) that
>> may help track it down.
>>
>> -K
>> On Jul 5, 2015 2:37 PM, "Larry Rosenman" <ler_at_lerctr.org> wrote:
>>
>>  I've gotten a couple of these:
>>>
>>> borg.lerctr.org dumped core - see /var/crash/vmcore.3
>>>
>>> Sun Jul  5 16:32:12 CDT 2015
>>>
>>> FreeBSD borg.lerctr.org 11.0-CURRENT FreeBSD 11.0-CURRENT #1 r285159:
>>> Sun
>>> Jul  5 09:35:33 CDT 2015     root_at_borg.lerctr.org:
>>> /usr/obj/usr/src/sys/LER
>>> amd64
>>>
>>> panic: general protection fault
>>>
>>> GNU gdb 6.1.1 [FreeBSD]
>>> Copyright 2004 Free Software Foundation, Inc.
>>> GDB is free software, covered by the GNU General Public License, and you
>>> are
>>> welcome to change it and/or distribute copies of it under certain
>>> conditions.
>>> Type "show copying" to see the conditions.
>>> There is absolutely no warranty for GDB.  Type "show warranty" for
>>> details.
>>> This GDB was configured as "amd64-marcel-freebsd"...
>>>
>>> Unread portion of the kernel message buffer:
>>>
>>>
>>> Fatal trap 9: general protection fault while in kernel mode
>>> cpuid = 0; apic id = 00
>>> instruction pointer     = 0x20:0xffffffff8083a15f
>>> stack pointer           = 0x28:0xfffffe100ca1d8a0
>>> frame pointer           = 0x28:0xfffffe100ca1d8e0
>>> code segment            = base 0x0, limit 0xfffff, type 0x1b
>>>                         = DPL 0, pres 1, long 1, def32 0, gran 1
>>> processor eflags        = interrupt enabled, resume, IOPL = 0
>>> current process         = 73971 (make)
>>> trap number             = 9
>>> panic: general protection fault
>>> cpuid = 0
>>> KDB: stack backtrace:
>>> db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame
>>> 0xfffffe100ca1d490
>>> vpanic() at vpanic+0x189/frame 0xfffffe100ca1d510
>>> panic() at panic+0x43/frame 0xfffffe100ca1d570
>>> trap_fatal() at trap_fatal+0x355/frame 0xfffffe100ca1d5d0
>>> trap() at trap+0x813/frame 0xfffffe100ca1d7e0
>>> calltrap() at calltrap+0x8/frame 0xfffffe100ca1d7e0
>>> --- trap 0x9, rip = 0xffffffff8083a15f, rsp = 0xfffffe100ca1d8a0, rbp =
>>> 0xfffffe100ca1d8e0 ---
>>> __mtx_lock_flags() at __mtx_lock_flags+0x4f/frame 0xfffffe100ca1d8e0
>>> doselwakeup() at doselwakeup+0xb5/frame 0xfffffe100ca1d920
>>> pipeselwakeup() at pipeselwakeup+0x46/frame 0xfffffe100ca1d940
>>> pipe_write() at pipe_write+0x13d2/frame 0xfffffe100ca1d9f0
>>> dofilewrite() at dofilewrite+0x8a/frame 0xfffffe100ca1da40
>>> kern_writev() at kern_writev+0x68/frame 0xfffffe100ca1da90
>>> sys_write() at sys_write+0x63/frame 0xfffffe100ca1dae0
>>> amd64_syscall() at amd64_syscall+0x25d/frame 0xfffffe100ca1dbf0
>>> Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe100ca1dbf0
>>> --- syscall (4, FreeBSD ELF64, sys_write), rip = 0x45ae8a, rsp =
>>> 0x7fffffffd328, rbp = 0x7fffffffd350 ---
>>> Uptime: 6h39m51s
>>> Dumping 3186 out of 64460
>>> MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91%
>>>
>>> Reading symbols from /boot/kernel/zfs.ko.symbols...done.
>>> Loaded symbols for /boot/kernel/zfs.ko.symbols
>>> Reading symbols from /boot/kernel/opensolaris.ko.symbols...done.
>>> Loaded symbols for /boot/kernel/opensolaris.ko.symbols
>>> Reading symbols from /boot/kernel/linux.ko.symbols...done.
>>> Loaded symbols for /boot/kernel/linux.ko.symbols
>>> Reading symbols from /boot/kernel/linux_common.ko.symbols...done.
>>> Loaded symbols for /boot/kernel/linux_common.ko.symbols
>>> Reading symbols from /boot/kernel/if_lagg.ko.symbols...done.
>>> Loaded symbols for /boot/kernel/if_lagg.ko.symbols
>>> Reading symbols from /boot/kernel/snd_envy24ht.ko.symbols...done.
>>> Loaded symbols for /boot/kernel/snd_envy24ht.ko.symbols
>>> Reading symbols from /boot/kernel/snd_spicds.ko.symbols...done.
>>> Loaded symbols for /boot/kernel/snd_spicds.ko.symbols
>>> Reading symbols from /boot/kernel/coretemp.ko.symbols...done.
>>> Loaded symbols for /boot/kernel/coretemp.ko.symbols
>>> Reading symbols from /boot/kernel/ichsmb.ko.symbols...done.
>>> Loaded symbols for /boot/kernel/ichsmb.ko.symbols
>>> Reading symbols from /boot/kernel/smbus.ko.symbols...done.
>>> Loaded symbols for /boot/kernel/smbus.ko.symbols
>>> Reading symbols from /boot/kernel/ichwd.ko.symbols...done.
>>> Loaded symbols for /boot/kernel/ichwd.ko.symbols
>>> Reading symbols from /boot/kernel/cpuctl.ko.symbols...done.
>>> Loaded symbols for /boot/kernel/cpuctl.ko.symbols
>>> Reading symbols from /boot/kernel/cryptodev.ko.symbols...done.
>>> Loaded symbols for /boot/kernel/cryptodev.ko.symbols
>>> Reading symbols from /boot/kernel/dtraceall.ko.symbols...done.
>>> Loaded symbols for /boot/kernel/dtraceall.ko.symbols
>>> Reading symbols from /boot/kernel/profile.ko.symbols...done.
>>> Loaded symbols for /boot/kernel/profile.ko.symbols
>>> Reading symbols from /boot/kernel/dtrace.ko.symbols...done.
>>> Loaded symbols for /boot/kernel/dtrace.ko.symbols
>>> Reading symbols from /boot/kernel/systrace_freebsd32.ko.symbols...done.
>>> Loaded symbols for /boot/kernel/systrace_freebsd32.ko.symbols
>>> Reading symbols from /boot/kernel/systrace.ko.symbols...done.
>>> Loaded symbols for /boot/kernel/systrace.ko.symbols
>>> Reading symbols from /boot/kernel/sdt.ko.symbols...done.
>>> Loaded symbols for /boot/kernel/sdt.ko.symbols
>>> Reading symbols from /boot/kernel/lockstat.ko.symbols...done.
>>> Loaded symbols for /boot/kernel/lockstat.ko.symbols
>>> Reading symbols from /boot/kernel/fasttrap.ko.symbols...done.
>>> Loaded symbols for /boot/kernel/fasttrap.ko.symbols
>>> Reading symbols from /boot/kernel/fbt.ko.symbols...done.
>>> Loaded symbols for /boot/kernel/fbt.ko.symbols
>>> Reading symbols from /boot/kernel/dtnfscl.ko.symbols...done.
>>> Loaded symbols for /boot/kernel/dtnfscl.ko.symbols
>>> Reading symbols from /boot/kernel/dtmalloc.ko.symbols...done.
>>> Loaded symbols for /boot/kernel/dtmalloc.ko.symbols
>>> Reading symbols from /boot/modules/vboxdrv.ko...done.
>>> Loaded symbols for /boot/modules/vboxdrv.ko
>>> Reading symbols from /boot/modules/nvidia.ko...done.
>>> Loaded symbols for /boot/modules/nvidia.ko
>>> Reading symbols from /boot/kernel/ipmi.ko.symbols...done.
>>> Loaded symbols for /boot/kernel/ipmi.ko.symbols
>>> Reading symbols from /boot/kernel/ipmi_linux.ko.symbols...done.
>>> Loaded symbols for /boot/kernel/ipmi_linux.ko.symbols
>>> Reading symbols from /boot/kernel/radeonkms.ko.symbols...done.
>>> Loaded symbols for /boot/kernel/radeonkms.ko.symbols
>>> Reading symbols from /boot/kernel/iicbb.ko.symbols...done.
>>> Loaded symbols for /boot/kernel/iicbb.ko.symbols
>>> Reading symbols from /boot/kernel/iicbus.ko.symbols...done.
>>> Loaded symbols for /boot/kernel/iicbus.ko.symbols
>>> Reading symbols from /boot/kernel/iic.ko.symbols...done.
>>> Loaded symbols for /boot/kernel/iic.ko.symbols
>>> Reading symbols from /boot/kernel/drm2.ko.symbols...done.
>>> Loaded symbols for /boot/kernel/drm2.ko.symbols
>>> Reading symbols from /boot/kernel/radeonkmsfw_R100_cp.ko.symbols...done.
>>> Loaded symbols for /boot/kernel/radeonkmsfw_R100_cp.ko.symbols
>>> Reading symbols from /boot/kernel/uhid.ko.symbols...done.
>>> Loaded symbols for /boot/kernel/uhid.ko.symbols
>>> Reading symbols from /boot/modules/vboxnetflt.ko...done.
>>> Loaded symbols for /boot/modules/vboxnetflt.ko
>>> Reading symbols from /boot/kernel/netgraph.ko.symbols...done.
>>> Loaded symbols for /boot/kernel/netgraph.ko.symbols
>>> Reading symbols from /boot/kernel/ng_ether.ko.symbols...done.
>>> Loaded symbols for /boot/kernel/ng_ether.ko.symbols
>>> Reading symbols from /boot/modules/vboxnetadp.ko...done.
>>> Loaded symbols for /boot/modules/vboxnetadp.ko
>>> #0  doadump (textdump=1) at pcpu.h:221
>>> 221     pcpu.h: No such file or directory.
>>>         in pcpu.h
>>> (kgdb) #0  doadump (textdump=1) at pcpu.h:221
>>> #1  0xffffffff80855d55 in kern_reboot (howto=260)
>>>     at /usr/src/sys/kern/kern_shutdown.c:447
>>> #2  0xffffffff80856348 in vpanic (fmt=<value optimized out>,
>>>     ap=<value optimized out>) at /usr/src/sys/kern/kern_shutdown.c:744
>>> #3  0xffffffff80856393 in panic (fmt=0x0)
>>>     at /usr/src/sys/kern/kern_shutdown.c:675
>>> #4  0xffffffff80b57275 in trap_fatal (frame=<value optimized out>,
>>>     eva=<value optimized out>) at /usr/src/sys/amd64/amd64/trap.c:853
>>> #5  0xffffffff80b56f03 in trap (frame=<value optimized out>)
>>>     at /usr/src/sys/amd64/amd64/trap.c:201
>>> #6  0xffffffff80b37142 in calltrap ()
>>>     at /usr/src/sys/amd64/amd64/exception.S:235
>>> #7  0xffffffff8083a15f in __mtx_lock_flags (c=0xdeadc0dedeadc116, opts=0,
>>>     file=0xffffffff80da2bc8 "/usr/src/sys/kern/sys_generic.c", line=1808)
>>>     at /usr/src/sys/kern/kern_mutex.c:211
>>> #8  0xffffffff808b1b25 in doselwakeup (sip=0xfffff8019bd5b0b8, pri=104)
>>>     at /usr/src/sys/kern/sys_generic.c:1808
>>> #9  0xffffffff808b4b86 in pipeselwakeup (cpipe=0xfffff8019bd5b000)
>>>     at /usr/src/sys/kern/sys_pipe.c:622
>>> #10 0xffffffff808b3892 in pipe_write (fp=0xfffff80027c3d280,
>>>     uio=<value optimized out>, active_cred=<value optimized out>,
>>> flags=1808,
>>>     td=0xfffff80736fda4c0) at /usr/src/sys/kern/sys_pipe.c:1314
>>> #11 0xffffffff808af7da in dofilewrite (td=0xfffff80736fda4c0, fd=1,
>>>     fp=0xfffff80027c3d280, auio=0xfffffe100ca1dab0,
>>>     offset=<value optimized out>, flags=0) at file.h:310
>>> #12 0xffffffff808af4e8 in kern_writev (td=0xfffff80736fda4c0, fd=1,
>>>     auio=0xfffffe100ca1dab0) at /usr/src/sys/kern/sys_generic.c:485
>>> #13 0xffffffff808af473 in sys_write (td=0xdeadc0dedeadc116,
>>>     uap=<value optimized out>) at /usr/src/sys/kern/sys_generic.c:398
>>> #14 0xffffffff80b57a6d in amd64_syscall (td=0xfffff80736fda4c0, traced=0)
>>>     at subr_syscall.c:133
>>> #15 0xffffffff80b3742b in Xfast_syscall ()
>>>     at /usr/src/sys/amd64/amd64/exception.S:395
>>> #16 0x000000000045ae8a in ?? ()
>>> Previous frame inner to this frame (corrupt stack?)
>>> Current language:  auto; currently minimal
>>> (kgdb)
>>>
>>> Ideas?
>>>
>>>
>>> FreeBSD borg.lerctr.org 11.0-CURRENT FreeBSD 11.0-CURRENT #1 r285159:
>>> Sun
>>> Jul  5 09:35:33 CDT 2015     root_at_borg.lerctr.org:
>>> /usr/obj/usr/src/sys/LER
>>> amd64 1100077 1100077
>>> --
>>> Larry Rosenman                     http://www.lerctr.org/~ler
>>> Phone: +1 214-642-9640                 E-Mail: ler_at_lerctr.org
>>> US Mail: 108 Turvey Cove, Hutto, TX 78634-5688
>>> _______________________________________________
>>> freebsd-current_at_freebsd.org mailing list
>>> http://lists.freebsd.org/mailman/listinfo/freebsd-current
>>> To unsubscribe, send any mail to "
>>> freebsd-current-unsubscribe_at_freebsd.org"
>>>
>>>  _______________________________________________
>> freebsd-current_at_freebsd.org mailing list
>> http://lists.freebsd.org/mailman/listinfo/freebsd-current
>> To unsubscribe, send any mail to "freebsd-current-unsubscribe_at_freebsd.org
>> "
>>
>
> --
> Larry Rosenman                     http://www.lerctr.org/~ler
> Phone: +1 214-642-9640                 E-Mail: ler_at_lerctr.org
> US Mail: 108 Turvey Cove, Hutto, TX 78634-5688
>
Received on Mon Jul 06 2015 - 20:43:03 UTC

This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:40:58 UTC