amd64 head -r349794 (under Hyper-V): "panic: spin lock held too long" during a buildworld buildkernel

From: Mark Millard <marklmi_at_yahoo.com>
Date: Sat, 6 Jul 2019 19:49:15 -0700
Looks like pmap_invalidate_range using smp_targeted_tlb_shootdown using _mtx_lock_spin_cookie.

I'll note that I had no trouble with -r349444 building world or kernel repeatedly, including
when I originally build -r349794 to upgrade.

The below is from my 2nd buildworld buildkernel under -r349794 (but the 2 builds were
not from scratch ones).


# more /var/crash/core.txt.2 
FBSDFSSD dumped core - see /var/crash/vmcore.2

Sun Jul  7 02:26:36 PDT 2019

FreeBSD FBSDFSSD 13.0-CURRENT FreeBSD 13.0-CURRENT #24 r349794M: Sun Jul  7 01:55:57 PDT 2019     markmi_at_FBSDFSSD:/usr/obj/amd64_clang/amd64.amd64/usr/src/amd64.amd64/sys/GENERIC-NODBG  amd64

panic: spin lock held too long

GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd"...

Unread portion of the kernel message buffer:
spin lock 0xffffffff829540e0 (smp rendezvous) held by 0xfffff80ae7ebb5a0 (tid 100669) too long
timeout stopping cpus
panic: spin lock held too long
cpuid = 15
time = 1562491248
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe00db5263c0
vpanic() at vpanic+0x19d/frame 0xfffffe00db526410
panic() at panic+0x43/frame 0xfffffe00db526470
_mtx_lock_indefinite_check() at _mtx_lock_indefinite_check+0x6d/frame 0xfffffe00db526480
_mtx_lock_spin_cookie() at _mtx_lock_spin_cookie+0xd5/frame 0xfffffe00db5264f0
smp_targeted_tlb_shootdown() at smp_targeted_tlb_shootdown+0x3de/frame 0xfffffe00db526560
pmap_invalidate_range() at pmap_invalidate_range+0x25c/frame 0xfffffe00db5265f0
vm_thread_stack_dispose() at vm_thread_stack_dispose+0x2c/frame 0xfffffe00db526640
thread_reap() at thread_reap+0x106/frame 0xfffffe00db526660
proc_reap() at proc_reap+0x788/frame 0xfffffe00db5266a0
proc_to_reap() at proc_to_reap+0x463/frame 0xfffffe00db5266f0
kern_wait6() at kern_wait6+0x34c/frame 0xfffffe00db526790
sys_wait4() at sys_wait4+0x78/frame 0xfffffe00db526980
amd64_syscall() at amd64_syscall+0x36e/frame 0xfffffe00db526ab0
fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe00db526ab0
--- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x80038f7fa, rsp = 0x7fffffffb168, rbp = 0x7fffffffb1b0 ---
KDB: enter: panic

Reading symbols from /boot/kernel/intpm.ko...Reading symbols from /usr/lib/debug//boot/kernel/intpm.ko.debug...done.
done.
Loaded symbols for /boot/kernel/intpm.ko
Reading symbols from /boot/kernel/smbus.ko...Reading symbols from /usr/lib/debug//boot/kernel/smbus.ko.debug...done.
done.
Loaded symbols for /boot/kernel/smbus.ko
Reading symbols from /boot/kernel/mac_ntpd.ko...Reading symbols from /usr/lib/debug//boot/kernel/mac_ntpd.ko.debug...done.
done.
Loaded symbols for /boot/kernel/mac_ntpd.ko
Reading symbols from /boot/kernel/imgact_binmisc.ko...Reading symbols from /usr/lib/debug//boot/kernel/imgact_binmisc.ko.debug...done.
done.
Loaded symbols for /boot/kernel/imgact_binmisc.ko
Reading symbols from /boot/kernel/filemon.ko...Reading symbols from /usr/lib/debug//boot/kernel/filemon.ko.debug...done.
done.
Loaded symbols for /boot/kernel/filemon.ko
#0  doadump (textdump=0) at src/sys/amd64/include/pcpu.h:246
246             __asm("movq %%gs:%P1,%0" : "=r" (td) : "n" (OFFSETOF_CURTHREAD));
(kgdb) #0  doadump (textdump=0) at src/sys/amd64/include/pcpu.h:246
#1  0xffffffff804a152b in db_dump (dummy=<value optimized out>, 
    dummy3=<value optimized out>, dummy4=<value optimized out>)
    at /usr/src/sys/ddb/db_command.c:575
#2  0xffffffff804a12f9 in db_command (cmd_table=<value optimized out>, 
    dopager=1) at /usr/src/sys/ddb/db_command.c:482
#3  0xffffffff804a1074 in db_command_loop ()
    at /usr/src/sys/ddb/db_command.c:535
#4  0xffffffff804a42cf in db_trap (type=<value optimized out>, 
    code=<value optimized out>) at /usr/src/sys/ddb/db_main.c:252
#5  0xffffffff80c502bc in kdb_trap (type=3, code=0, tf=<value optimized out>)
    at /usr/src/sys/kern/subr_kdb.c:692
#6  0xffffffff810e1e2c in trap (frame=0xfffffe00db5262f0)
    at /usr/src/sys/amd64/amd64/trap.c:621
#7  0xffffffff810bb8b5 in calltrap ()
    at /usr/src/sys/amd64/amd64/exception.S:232
#8  0xffffffff80c4f9cb in kdb_enter (why=0xffffffff8134eb81 "panic", 
    msg=<value optimized out>) at src/sys/amd64/include/cpufunc.h:65
#9  0xffffffff80c0226a in vpanic (fmt=<value optimized out>, 
    ap=<value optimized out>) at /usr/src/sys/kern/kern_shutdown.c:894
#10 0xffffffff80c020a3 in panic (fmt=<value optimized out>)
    at /usr/src/sys/kern/kern_shutdown.c:832
#11 0xffffffff80be128d in _mtx_lock_indefinite_check (m=<value optimized out>, 
    ldap=<value optimized out>) at /usr/src/sys/kern/kern_mutex.c:1222
#12 0xffffffff80be0dd5 in _mtx_lock_spin_cookie (c=0xffffffff829540f8, 
    v=<value optimized out>) at /usr/src/sys/kern/kern_mutex.c:748
#13 0xffffffff8125b32e in smp_targeted_tlb_shootdown (mask=
      {__bits = 0xfffffe00db526570}, vector=246, pmap=0xffffffff82a682f8, 
    addr1=18446741878369513472, addr2=18446741878369529856)
    at /usr/src/sys/x86/x86/mp_x86.c:1671
#14 0xffffffff810cae6c in pmap_invalidate_range (pmap=0xffffffff82a682f8, 
    sva=18446741878369513472, eva=18446741878369529856)
    at /usr/src/sys/amd64/amd64/pmap.c:2425
#15 0xffffffff80f474fc in vm_thread_stack_dispose (ksobj=0xfffff80123acdb00, 
    ks=18446741878369513472, pages=4) at /usr/src/sys/vm/vm_glue.c:404
#16 0xffffffff80c16796 in thread_reap () at /usr/src/sys/kern/kern_thread.c:449
#17 0xffffffff80bbb628 in proc_reap (td=<value optimized out>, 
    p=0xfffff8010b99ea60, status=<value optimized out>, options=48)
    at /usr/src/sys/kern/kern_exit.c:969
#18 0xffffffff80bbbd03 in proc_to_reap (td=0xfffff80a0d384000, 
    p=0xfffff8010b99ea60, idtype=<value optimized out>, 
    id=<value optimized out>, status=<value optimized out>, 
    options=<value optimized out>, wrusage=0x0, siginfo=0x0, check_only=0)
    at /usr/src/sys/kern/kern_exit.c:1130
#19 0xffffffff80bbaa2c in kern_wait6 (td=0xfffff80a0d384000, idtype=P_ALL, 
    id=0, status=0xfffffe00db52695c, options=48, wrusage=0x0, siginfo=0x0)
    at /usr/src/sys/kern/kern_exit.c:1256
#20 0xffffffff80bba488 in sys_wait4 (td=0xfffff80a0d384000, 
    uap=0xfffff80a0d3843c8) at /usr/src/sys/kern/kern_exit.c:1171
#21 0xffffffff810e2e7e in amd64_syscall (td=0xfffff80a0d384000, traced=0)
    at src/sys/amd64/amd64/../../kern/subr_syscall.c:135
#22 0xffffffff810bc19d in fast_syscall_common ()
    at /usr/src/sys/amd64/amd64/exception.S:504
#23 0x000000080038f7fa in ?? ()
Previous frame inner to this frame (corrupt stack?)
Current language:  auto; currently minimal
(kgdb) 

. . .

FreeBSD 13.0-CURRENT #24 r349794M: Sun Jul  7 01:55:57 PDT 2019
    markmi_at_FBSDFSSD:/usr/obj/amd64_clang/amd64.amd64/usr/src/amd64.amd64/sys/GENERIC-NODBG amd64
FreeBSD clang version 8.0.1 (branches/release_80 363030) (based on LLVM 8.0.1)
SRAT: Ignoring memory at addr 0x1428000000
SRAT: Ignoring memory at addr 0x8894000000
VT(vga): text 80x25
Hyper-V Version: 10.0.18362 [SP0]
  Features=0x2e7f<VPRUNTIME,TMREFCNT,SYNIC,SYNTM,APIC,HYPERCALL,VPINDEX,REFTSC,IDLE,TMFREQ>
  PM Features=0x0 [C2]
  Features3=0xbed7b2<DEBUG,XMMHC,IDLE,NUMA,TMFREQ,SYNCMC,CRASH,NPIEP>
Timecounter "Hyper-V" frequency 10000000 Hz quality 2000
CPU: AMD Ryzen Threadripper 1950X 16-Core Processor  (3393.67-MHz K8-class CPU)
  Origin="AuthenticAMD"  Id=0x800f11  Family=0x17  Model=0x1  Stepping=1
  Features=0x1783fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE,SSE2,HTT>
  Features2=0xfed83203<SSE3,PCLMULQDQ,SSSE3,FMA,CX16,SSE4.1,SSE4.2,MOVBE,POPCNT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND,HV>
  AMD Features=0x2e500800<SYSCALL,NX,MMX+,FFXSR,Page1GB,RDTSCP,LM>
  AMD Features2=0x4003f3<LAHF,CMP,CR8,ABM,SSE4A,MAS,Prefetch,OSVW,Topology>
  Structured Extended Features=0x209c01a9<FSGSBASE,BMI1,AVX2,SMEP,BMI2,RDSEED,ADX,SMAP,CLFLUSHOPT,SHA>
  XSAVE Features=0xf<XSAVEOPT,XSAVEC,XINUSE,XSAVES>
  AMD Extended Feature Extensions ID EBX=0x2001004<XSaveErPtr,IBPB,VIRT_SSBD>
Hypervisor: Origin = "Microsoft Hv"

. . .

===
Mark Millard
marklmi at yahoo.com
( dsl-only.net went
away in early 2018-Mar)
Received on Sun Jul 07 2019 - 01:59:05 UTC

This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:41:21 UTC