Crash loading dtraceall

From: Larry Rosenman <ler_at_FreeBSD.org>
Date: Wed, 8 May 2019 15:52:45 -0500
Greetings,

    Somewhere between r346483 and r347241 loading dtraceall causes a
    crash.  I have the cores and kernels. 

    It's hard for me to bisect more than this, as the box is remote. 

    What more do you need?  (this dump is fropm r347355). 



[ler_at_oldtbh2.lerctr.org:/var/crash] $ more core.txt.5
oldtbh2.lerctr.org dumped core - see /var/crash/vmcore.5

Wed May  8 15:46:58 CDT 2019

FreeBSD oldtbh2.lerctr.org 13.0-CURRENT FreeBSD 13.0-CURRENT r347355 LER-MINIMAL  amd64

panic: page fault

GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd"...

Unread portion of the kernel message buffer:


Fatal trap 12: page fault while in kernel mode
cpuid = 3; apic id = 03
fault virtual address   = 0x0
fault code              = supervisor read data  , page not present
instruction pointer     = 0x20:0xffffffff825cb5ea
stack pointer           = 0x28:0xfffffe00d74d7d90
frame pointer           = 0x28:0xfffffe00d74d7df0
code segment            = base 0x0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags        = interrupt enabled, resume, IOPL = 0
current process         = 1531 (kldload)
trap number             = 12
panic: page fault
cpuid = 3
time = 1557348131
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe00d74d7a40
vpanic() at vpanic+0x19d/frame 0xfffffe00d74d7a90
panic() at panic+0x43/frame 0xfffffe00d74d7af0
trap_fatal() at trap_fatal+0x394/frame 0xfffffe00d74d7b50
trap_pfault() at trap_pfault+0x49/frame 0xfffffe00d74d7bb0
trap() at trap+0x2b4/frame 0xfffffe00d74d7cc0
calltrap() at calltrap+0x8/frame 0xfffffe00d74d7cc0
--- trap 0xc, rip = 0xffffffff825cb5ea, rsp = 0xfffffe00d74d7d90, rbp = 0xfffffe00d74d7df0 ---
fbt_provide_module_function() at fbt_provide_module_function+0x7a/frame 0xfffffe00d74d7df0
link_elf_each_function_nameval() at link_elf_each_function_nameval+0xf7/frame 0xfffffe00d74d7e40
fbt_provide_module() at fbt_provide_module+0xde/frame 0xfffffe00d74d8270
fbt_linker_file_cb() at fbt_linker_file_cb+0x12/frame 0xfffffe00d74d8280
linker_file_foreach() at linker_file_foreach+0x52/frame 0xfffffe00d74d82b0
linker_load_module() at linker_load_module+0xbd8/frame 0xfffffe00d74d85e0
linker_load_dependencies() at linker_load_dependencies+0x2fd/frame 0xfffffe00d74d8630
link_elf_load_file() at link_elf_load_file+0x105e/frame 0xfffffe00d74d86f0
linker_load_module() at linker_load_module+0x9ef/frame 0xfffffe00d74d8a20
kern_kldload() at kern_kldload+0xa7/frame 0xfffffe00d74d8a60
sys_kldload() at sys_kldload+0x5b/frame 0xfffffe00d74d8a90
amd64_syscall() at amd64_syscall+0x25c/frame 0xfffffe00d74d8bb0
fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe00d74d8bb0
--- syscall (304, FreeBSD ELF64, sys_kldload), rip = 0x8002de43a, rsp = 0x7fffffffe658, rbp = 0x7fffffffebd0 ---
Uptime: 2m40s
Dumping 2258 out of 64482 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91%

Reading symbols from /boot/kernel/zfs.ko...Reading symbols from /usr/lib/debug//boot/kernel/zfs.ko.debug...done.
done.
Loaded symbols for /boot/kernel/zfs.ko
Reading symbols from /boot/kernel/krpc.ko...Reading symbols from /usr/lib/debug//boot/kernel/krpc.ko.debug...done.
done.
Loaded symbols for /boot/kernel/krpc.ko
Reading symbols from /boot/kernel/opensolaris.ko...Reading symbols from /usr/lib/debug//boot/kernel/opensolaris.ko.debug...done.
done.
Loaded symbols for /boot/kernel/opensolaris.ko
Reading symbols from /boot/kernel/if_em.ko...Reading symbols from /usr/lib/debug//boot/kernel/if_em.ko.debug...done.
done.
Loaded symbols for /boot/kernel/if_em.ko
Reading symbols from /boot/kernel/iflib.ko...Reading symbols from /usr/lib/debug//boot/kernel/iflib.ko.debug...done.
done.
Loaded symbols for /boot/kernel/iflib.ko
Reading symbols from /boot/kernel/if_bce.ko...Reading symbols from /usr/lib/debug//boot/kernel/if_bce.ko.debug...done.
done.
Loaded symbols for /boot/kernel/if_bce.ko
Reading symbols from /boot/kernel/miibus.ko...Reading symbols from /usr/lib/debug//boot/kernel/miibus.ko.debug...done.
done.
Loaded symbols for /boot/kernel/miibus.ko
Reading symbols from /boot/kernel/coretemp.ko...Reading symbols from /usr/lib/debug//boot/kernel/coretemp.ko.debug...done.
done.
Loaded symbols for /boot/kernel/coretemp.ko
Reading symbols from /boot/kernel/filemon.ko...Reading symbols from /usr/lib/debug//boot/kernel/filemon.ko.debug...done.
done.
Loaded symbols for /boot/kernel/filemon.ko
Reading symbols from /boot/kernel/fuse.ko...Reading symbols from /usr/lib/debug//boot/kernel/fusefs.ko.debug...done.
done.
Loaded symbols for /boot/kernel/fuse.ko
Reading symbols from /boot/kernel/linux.ko...Reading symbols from /usr/lib/debug//boot/kernel/linux.ko.debug...done.
done.
Loaded symbols for /boot/kernel/linux.ko
Reading symbols from /boot/kernel/linux_common.ko...Reading symbols from /usr/lib/debug//boot/kernel/linux_common.ko.debug...done.
done.
Loaded symbols for /boot/kernel/linux_common.ko
Reading symbols from /boot/kernel/ichsmb.ko...Reading symbols from /usr/lib/debug//boot/kernel/ichsmb.ko.debug...done.
done.
Loaded symbols for /boot/kernel/ichsmb.ko
Reading symbols from /boot/kernel/smbus.ko...Reading symbols from /usr/lib/debug//boot/kernel/smbus.ko.debug...done.
done.
Loaded symbols for /boot/kernel/smbus.ko
Reading symbols from /boot/kernel/ichwd.ko...Reading symbols from /usr/lib/debug//boot/kernel/ichwd.ko.debug...done.
done.
Loaded symbols for /boot/kernel/ichwd.ko
Reading symbols from /boot/kernel/cpuctl.ko...Reading symbols from /usr/lib/debug//boot/kernel/cpuctl.ko.debug...done.
done.
Loaded symbols for /boot/kernel/cpuctl.ko
Reading symbols from /boot/kernel/crypto.ko...Reading symbols from /usr/lib/debug//boot/kernel/crypto.ko.debug...done.
done.
Loaded symbols for /boot/kernel/crypto.ko
Reading symbols from /boot/kernel/cryptodev.ko...Reading symbols from /usr/lib/debug//boot/kernel/cryptodev.ko.debug...done.
done.
Loaded symbols for /boot/kernel/cryptodev.ko
Reading symbols from /boot/kernel/ipmi.ko...Reading symbols from /usr/lib/debug//boot/kernel/ipmi.ko.debug...done.
done.
Loaded symbols for /boot/kernel/ipmi.ko
Reading symbols from /boot/kernel/hwpmc.ko...Reading symbols from /usr/lib/debug//boot/kernel/hwpmc.ko.debug...done.
done.
Loaded symbols for /boot/kernel/hwpmc.ko
Reading symbols from /boot/kernel/tcp_rack.ko...Reading symbols from /usr/lib/debug//boot/kernel/tcp_rack.ko.debug...done.
done.
Loaded symbols for /boot/kernel/tcp_rack.ko
Reading symbols from /boot/kernel/mfip.ko...Reading symbols from /usr/lib/debug//boot/kernel/mfip.ko.debug...done.
done.
Loaded symbols for /boot/kernel/mfip.ko
Reading symbols from /boot/kernel/cam.ko...Reading symbols from /usr/lib/debug//boot/kernel/cam.ko.debug...done.
done.
Loaded symbols for /boot/kernel/cam.ko
Reading symbols from /boot/kernel/usb.ko...Reading symbols from /usr/lib/debug//boot/kernel/usb.ko.debug...done.
done.
Loaded symbols for /boot/kernel/usb.ko
Reading symbols from /boot/kernel/uhci.ko...Reading symbols from /usr/lib/debug//boot/kernel/uhci.ko.debug...done.
done.
Loaded symbols for /boot/kernel/uhci.ko
Reading symbols from /boot/kernel/xhci.ko...Reading symbols from /usr/lib/debug//boot/kernel/xhci.ko.debug...done.
done.
Loaded symbols for /boot/kernel/xhci.ko
Reading symbols from /boot/kernel/ehci.ko...Reading symbols from /usr/lib/debug//boot/kernel/ehci.ko.debug...done.
done.
Loaded symbols for /boot/kernel/ehci.ko
Reading symbols from /boot/kernel/ohci.ko...Reading symbols from /usr/lib/debug//boot/kernel/ohci.ko.debug...done.
done.
Loaded symbols for /boot/kernel/ohci.ko
Reading symbols from /boot/kernel/ukbd.ko...Reading symbols from /usr/lib/debug//boot/kernel/ukbd.ko.debug...done.
done.
Loaded symbols for /boot/kernel/ukbd.ko
Reading symbols from /boot/kernel/ums.ko...Reading symbols from /usr/lib/debug//boot/kernel/ums.ko.debug...done.
done.
Loaded symbols for /boot/kernel/ums.ko
Reading symbols from /boot/kernel/usb_quirk.ko...Reading symbols from /usr/lib/debug//boot/kernel/usb_quirk.ko.debug...done.
done.
Loaded symbols for /boot/kernel/usb_quirk.ko
Reading symbols from /boot/kernel/usb_template.ko...Reading symbols from /usr/lib/debug//boot/kernel/usb_template.ko.debug...done.
done.
Loaded symbols for /boot/kernel/usb_template.ko
Reading symbols from /boot/kernel/efirt.ko...Reading symbols from /usr/lib/debug//boot/kernel/efirt.ko.debug...done.
done.
Loaded symbols for /boot/kernel/efirt.ko
Reading symbols from /boot/kernel/nfscl.ko...Reading symbols from /usr/lib/debug//boot/kernel/nfscl.ko.debug...done.
done.
Loaded symbols for /boot/kernel/nfscl.ko
Reading symbols from /boot/kernel/nfscommon.ko...Reading symbols from /usr/lib/debug//boot/kernel/nfscommon.ko.debug...done.
done.
Loaded symbols for /boot/kernel/nfscommon.ko
Reading symbols from /boot/kernel/nfssvc.ko...Reading symbols from /usr/lib/debug//boot/kernel/nfssvc.ko.debug...done.
done.
Loaded symbols for /boot/kernel/nfssvc.ko
Reading symbols from /boot/kernel/nfslock.ko...Reading symbols from /usr/lib/debug//boot/kernel/nfslock.ko.debug...done.
done.
Loaded symbols for /boot/kernel/nfslock.ko
Reading symbols from /boot/kernel/nfsd.ko...Reading symbols from /usr/lib/debug//boot/kernel/nfsd.ko.debug...done.
done.
Loaded symbols for /boot/kernel/nfsd.ko
Reading symbols from /boot/kernel/nfslockd.ko...Reading symbols from /usr/lib/debug//boot/kernel/nfslockd.ko.debug...done.
done.
Loaded symbols for /boot/kernel/nfslockd.ko
Reading symbols from /boot/kernel/ataintel.ko...Reading symbols from /usr/lib/debug//boot/kernel/ataintel.ko.debug...done.
done.
Loaded symbols for /boot/kernel/ataintel.ko
Reading symbols from /boot/kernel/ata.ko...Reading symbols from /usr/lib/debug//boot/kernel/ata.ko.debug...done.
done.
Loaded symbols for /boot/kernel/ata.ko
Reading symbols from /boot/kernel/atapci.ko...Reading symbols from /usr/lib/debug//boot/kernel/atapci.ko.debug...done.
done.
Loaded symbols for /boot/kernel/atapci.ko
Reading symbols from /boot/kernel/geom_label.ko...Reading symbols from /usr/lib/debug//boot/kernel/geom_label.ko.debug...done.
done.
Loaded symbols for /boot/kernel/geom_label.ko
Reading symbols from /boot/kernel/linux64.ko...Reading symbols from /usr/lib/debug//boot/kernel/linux64.ko.debug...done.
done.
Loaded symbols for /boot/kernel/linux64.ko
Reading symbols from /boot/kernel/smbios.ko...Reading symbols from /usr/lib/debug//boot/kernel/smbios.ko.debug...done.
done.
Loaded symbols for /boot/kernel/smbios.ko
Reading symbols from /boot/kernel/mac_ntpd.ko...Reading symbols from /usr/lib/debug//boot/kernel/mac_ntpd.ko.debug...done.
done.
Loaded symbols for /boot/kernel/mac_ntpd.ko
Reading symbols from /boot/kernel/netgraph.ko...Reading symbols from /usr/lib/debug//boot/kernel/netgraph.ko.debug...done.
done.
Loaded symbols for /boot/kernel/netgraph.ko
Reading symbols from /boot/kernel/ng_netflow.ko...Reading symbols from /usr/lib/debug//boot/kernel/ng_netflow.ko.debug...done.
done.
Loaded symbols for /boot/kernel/ng_netflow.ko
Reading symbols from /boot/kernel/ng_ksocket.ko...Reading symbols from /usr/lib/debug//boot/kernel/ng_ksocket.ko.debug...done.
done.
Loaded symbols for /boot/kernel/ng_ksocket.ko
Reading symbols from /boot/kernel/ng_ether.ko...Reading symbols from /usr/lib/debug//boot/kernel/ng_ether.ko.debug...done.
done.
Loaded symbols for /boot/kernel/ng_ether.ko
Reading symbols from /boot/kernel/ng_socket.ko...Reading symbols from /usr/lib/debug//boot/kernel/ng_socket.ko.debug...done.
done.
Loaded symbols for /boot/kernel/ng_socket.ko
Reading symbols from /boot/kernel/dtraceall.ko...Reading symbols from /usr/lib/debug//boot/kernel/dtraceall.ko.debug...done.
done.
Loaded symbols for /boot/kernel/dtraceall.ko
Reading symbols from /boot/kernel/dtrace.ko...Reading symbols from /usr/lib/debug//boot/kernel/dtrace.ko.debug...done.
done.
Loaded symbols for /boot/kernel/dtrace.ko
Reading symbols from /boot/kernel/dtmalloc.ko...Reading symbols from /usr/lib/debug//boot/kernel/dtmalloc.ko.debug...done.
done.
Loaded symbols for /boot/kernel/dtmalloc.ko
Reading symbols from /boot/kernel/fbt.ko...Reading symbols from /usr/lib/debug//boot/kernel/fbt.ko.debug...done.
done.
Loaded symbols for /boot/kernel/fbt.ko
#0  doadump (textdump=1) at src/sys/amd64/include/pcpu.h:241
241             __asm("movq %%gs:%P1,%0" : "=r" (td) : "n" (OFFSETOF_CURTHREAD));
(kgdb) #0  doadump (textdump=1) at src/sys/amd64/include/pcpu.h:241
#1  0xffffffff80496320 in kern_reboot (howto=260)
    at /usr/src/sys/kern/kern_shutdown.c:470
#2  0xffffffff80496799 in vpanic (fmt=<value optimized out>,
    ap=<value optimized out>) at /usr/src/sys/kern/kern_shutdown.c:896
#3  0xffffffff804964d3 in panic (fmt=<value optimized out>)
    at /usr/src/sys/kern/kern_shutdown.c:823
#4  0xffffffff80767314 in trap_fatal (frame=0xfffffe00d74d7cd0, eva=0)
    at /usr/src/sys/amd64/amd64/trap.c:946
#5  0xffffffff80767379 in trap_pfault (frame=0xfffffe00d74d7cd0, usermode=0)
    at src/sys/amd64/include/pcpu.h:241
#6  0xffffffff80766964 in trap (frame=0xfffffe00d74d7cd0)
    at /usr/src/sys/amd64/amd64/trap.c:441
#7  0xffffffff80740805 in calltrap ()
    at /usr/src/sys/amd64/amd64/exception.S:232
#8  0xffffffff825cb5ea in fbt_provide_module_function (lf=0xfffff800020ff000,
    symindx=30763, symval=0xfffffe00d74d7e00, opaque=0xfffffe00d74d7e50)
    at /usr/src/sys/cddl/dev/fbt/x86/fbt_isa.c:190
#9  0xffffffff804bf8f7 in link_elf_each_function_nameval (
    file=0xfffff800020ff000,
    callback=0xffffffff825cb570 <fbt_provide_module_function>,
    opaque=0xfffffe00d74d7e50) at /usr/src/sys/kern/link_elf.c:1513
#10 0xffffffff825ca33e in fbt_provide_module (arg=<value optimized out>,
    lf=0xfffff800020ff000) at /usr/src/sys/cddl/dev/fbt/fbt.c:204
#11 0xffffffff825ca242 in fbt_linker_file_cb (lf=<value optimized out>,
    arg=<value optimized out>) at /usr/src/sys/cddl/dev/fbt/fbt.c:1103
#12 0xffffffff8046d772 in linker_file_foreach (
    predicate=0xffffffff825ca230 <fbt_linker_file_cb>, context=0x0)
    at /usr/src/sys/kern/kern_linker.c:594
#13 0xffffffff8046cb58 in linker_load_module (kldname=<value optimized out>,
    modname=0xffffffff81d792ae "fbt", parent=<value optimized out>,
    verinfo=<value optimized out>, lfpp=0x0)
    at /usr/src/sys/kern/kern_linker.c:236
#14 0xffffffff8046f1bd in linker_load_dependencies (lf=0xfffff8002389a400)
    at /usr/src/sys/kern/kern_linker.c:2200
#15 0xffffffff80797f3e in link_elf_load_file (cls=<value optimized out>,
    filename=0xfffff80003d592c0 "/boot/kernel/dtraceall.ko",
    result=0xfffffe00d74d8898) at /usr/src/sys/kern/link_elf_obj.c:1010
#16 0xffffffff8046c96f in linker_load_module (kldname=<value optimized out>,
    modname=0xfffff800231a7800 "dtraceall", parent=<value optimized out>,
    verinfo=<value optimized out>, lfpp=0xfffffe00d74d8a38) at linker_if.h:180
#17 0xffffffff8046e297 in kern_kldload (td=0xfffff80114df9000,
    file=<value optimized out>, fileid=0xfffffe00d74d8a74)
    at /usr/src/sys/kern/kern_linker.c:1089
#18 0xffffffff8046e35b in sys_kldload (td=0xfffff80114df9000,
    uap=<value optimized out>) at /usr/src/sys/kern/kern_linker.c:1115
#19 0xffffffff80767ddc in amd64_syscall (td=0xfffff80114df9000, traced=0)
    at src/sys/amd64/amd64/../../kern/subr_syscall.c:135
#20 0xffffffff807410ed in fast_syscall_common ()
    at /usr/src/sys/amd64/amd64/exception.S:504
#21 0x00000008002de43a in ?? ()
Previous frame inner to this frame (corrupt stack?)
Current language:  auto; currently minimal
(kgdb)
-- 
Larry Rosenman                         https://people.FreeBSD.org/~ler/
Phone: +1 214-642-9640                 E-Mail: ler_at_FreeBSD.org
US Mail: 5708 Sabbia Drive, Round Rock, TX 78665-2106

Received on Wed May 08 2019 - 18:52:47 UTC

This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:41:20 UTC