recent 13.0-CURRENT double panic near start of poudriere run

From: Don Lewis <truckman_at_FreeBSD.org>
Date: Tue, 13 Nov 2018 23:05:23 -0800 (PST)
I've been doing a bunch of packing building today on my Ryzen box and
just got this kernel panic near the start of a poudriere run:

13.0-CURRENT FreeBSD 13.0-CURRENT r340358 GENERIC  amd64

[00:00:08] Processing PRIORITY_BOOST
[00:00:08] Balancing pool
[00:00:08] Recording filesystem state for prepkg... done
[00:00:09] Building 315 packages using 16 builders
[00:00:09] Starting/Cloning builders
[ system paniced and dropped into ddb ]

# /usr/local/bin/kgdb /boot/kernel/kernel /var/crash/vmcore.3
GNU gdb (GDB) 8.2 [GDB v8.2 for FreeBSD]
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-portbld-freebsd13.0".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /boot/kernel/kernel...Reading symbols from /usr/lib/debug//boot/kernel/kernel.debug...done.
done.

Unread portion of the kernel message buffer:
timeout stopping cpus
panic: mutex sched lock 2 recursed at /usr/src/sys/kern/kern_synch.c:390
cpuid = 2
time = 1542176879
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe00779645f0
vpanic() at vpanic+0x1a3/frame 0xfffffe0077964650
panic() at panic+0x43/frame 0xfffffe00779646b0
__mtx_assert() at __mtx_assert+0x69/frame 0xfffffe00779646c0
mi_switch() at mi_switch+0x42/frame 0xfffffe00779646f0
critical_exit_preempt() at critical_exit_preempt+0x66/frame 0xfffffe0077964710
lapic_handle_timer() at lapic_handle_timer+0xe4/frame 0xfffffe0077964750
Xtimerint() at Xtimerint+0xae/frame 0xfffffe0077964750
--- interrupt, rip = 0xffffffff80bf7860, rsp = 0xfffffe0077964820, rbp = 0xfffffe0077964840 ---
generic_stop_cpus() at generic_stop_cpus+0x170/frame 0xfffffe0077964840
stop_cpus_hard() at stop_cpus_hard+0x22/frame 0xfffffe0077964870
vpanic() at vpanic+0xb0/frame 0xfffffe00779648f0
panic() at panic+0x43/frame 0xfffffe0077964950
mi_switch() at mi_switch+0x1ff/frame 0xfffffe0077964980
critical_exit_preempt() at critical_exit_preempt+0x66/frame 0xfffffe00779649a0
sched_idletd() at sched_idletd+0x517/frame 0xfffffe0077964a70
fork_exit() at fork_exit+0x84/frame 0xfffffe0077964ab0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0077964ab0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
KDB: enter: panic

__curthread () at ./machine/pcpu.h:230
230		__asm("movq %%gs:%P1,%0" : "=r" (td) : "n" (OFFSETOF_CURTHREAD));

(kgdb) bt
#0  __curthread () at ./machine/pcpu.h:230
#1  doadump (textdump=-2118704160) at /usr/src/sys/kern/kern_shutdown.c:371
#2  0xffffffff804659bc in db_fncall_generic (addr=<optimized out>, 
    rv=<optimized out>, nargs=<optimized out>, args=<optimized out>)
    at /usr/src/sys/ddb/db_command.c:609
#3  db_fncall (dummy1=<optimized out>, dummy2=<optimized out>, 
    dummy3=<optimized out>, dummy4=<optimized out>)
    at /usr/src/sys/ddb/db_command.c:657
#4  0xffffffff804654f9 in db_command (last_cmdp=<optimized out>, 
    cmd_table=<optimized out>, dopager=<optimized out>)
    at /usr/src/sys/ddb/db_command.c:481
#5  0xffffffff80465274 in db_command_loop ()
    at /usr/src/sys/ddb/db_command.c:534
#6  0xffffffff8046848f in db_trap (type=<optimized out>, code=<optimized out>)
    at /usr/src/sys/ddb/db_main.c:252
#7  0xffffffff80be7925 in kdb_trap (type=3, code=0, tf=<optimized out>)
    at /usr/src/sys/kern/subr_kdb.c:692
#8  0xffffffff81074946 in trap (frame=0xfffffe0077964520)
    at /usr/src/sys/amd64/amd64/trap.c:619
#9  <signal handler called>
#10 kdb_enter (why=0xffffffff813065f5 "panic", msg=<optimized out>)
    at /usr/src/sys/kern/subr_kdb.c:479
#11 0xffffffff80b9f1f0 in vpanic (fmt=<optimized out>, ap=0xfffffe0077964690)
--Type <RET> for more, q to quit, c to continue without paging--c
    at /usr/src/sys/kern/kern_shutdown.c:866
#12 0xffffffff80b9ef93 in panic (fmt=0xffffffff81e951d8 <cnputs_mtx> "\304\271,\201\377\377\377\377") at /usr/src/sys/kern/kern_shutdown.c:804
#13 0xffffffff80b7ea79 in __mtx_assert (c=<optimized out>, what=<optimized out>, file=0xfffffe00779644e0 "", line=128) at /usr/src/sys/kern/kern_mutex.c:1064
#14 0xffffffff80baa6d2 in mi_switch (flags=1544, newtd=0x0) at /usr/src/sys/kern/kern_synch.c:390
#15 0xffffffff80ba7726 in critical_exit_preempt () at /usr/src/sys/kern/kern_switch.c:243
#16 0xffffffff811ea434 in critical_exit () at /usr/src/sys/sys/systm.h:266
#17 lapic_handle_timer (frame=0xfffffe0077964760) at /usr/src/sys/x86/x86/local_apic.c:1335
#18 <signal handler called>
#19 ia32_pause () at ./machine/cpufunc.h:348
#20 generic_stop_cpus (map=..., type=255) at /usr/src/sys/kern/subr_smp.c:280
#21 0xffffffff80bf78f2 in stop_cpus_hard (map=...) at /usr/src/sys/kern/subr_smp.c:308
#22 0xffffffff80b9f0e0 in vpanic (fmt=0xffffffff81210702 "mi_switch: switch in a critical section", ap=0xfffffe0077964930) at /usr/src/sys/kern/kern_shutdown.c:828
#23 0xffffffff80b9ef93 in panic (fmt=0x4400 <error: Cannot access memory at address 0x4400>) at /usr/src/sys/kern/kern_shutdown.c:804
#24 0xffffffff80baa88f in mi_switch (flags=1544, newtd=0x0) at /usr/src/sys/kern/kern_synch.c:396
#25 0xffffffff80ba7726 in critical_exit_preempt () at /usr/src/sys/kern/kern_switch.c:243
#26 0xffffffff80bcf2a7 in sched_idletd (dummy=<optimized out>) at /usr/src/sys/kern/sched_ule.c:2829
#27 0xffffffff80b5f064 in fork_exit (callout=0xffffffff80bced90 <sched_idletd>, arg=0x0, frame=0xfffffe0077964ac0) at /usr/src/sys/kern/kern_fork.c:1057
#28 <signal handler called>

It actually appears to be a double panic, with the original panic being:
  mi_switch: switch in a critical section
Received on Wed Nov 14 2018 - 06:05:26 UTC

This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:41:19 UTC