Hi everyone, I am currently experiencing semi-regular kernel crashes on my FreeBSD 12-current machine. I am new to kernel debugging, and hoping someone can have a look at the debugging output below to point me in the direction of what the problem might be. My machine is a FreeNAS-mini from iXsystems which I have formatted and installed stock FreeBSD onto. My kernel is the default generic-nodebug with the VIMAGE options added. $ uname -a FreeBSD freenas.woods.am 12.0-CURRENT FreeBSD 12.0-CURRENT #0 r305311M: Sat Sep 3 12:29:01 AWST 2016 woodsb02_at_freenas.woods.am:/usr/obj/usr/src/sys/GENERIC-NODEBUG-VIMAGE amd64 $ cat /usr/src/sys/amd64/conf/GENERIC-NODEBUG-VIMAGE # SPARTICUS -- WITNESS and INVARIANTS free kernel configuration file # for FreeBSD/amd64 include GENERIC-NODEBUG ident GENERIC-NODEBUG-VIMAGE #nooptions SCTP # Stream Control Transmission Protocol options VIMAGE # VNET/Vimage support Output from kernel crash dump debug with kgdb below: /usr/obj/usr/src/sys/GENERIC-NODEBUG-VIMAGE)# kgdb kernel.debug /var/crash/vmcore.last GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "amd64-marcel-freebsd"... Unread portion of the kernel message buffer: kernel trap 12 with interrupts disabled Fatal trap 12: page fault while in kernel mode cpuid = 2; apic id = 04 fault virtual address = 0x30 fault code = supervisor read data, page not present instruction pointer = 0x20:0xffffffff80b4d91c stack pointer = 0x28:0xfffffe046813a440 frame pointer = 0x28:0xfffffe046813a470 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = resume, IOPL = 0 current process = 33487 (sh) Uptime: 15m16s Dumping 1664 out of 16338 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91% Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/zfs.ko...Reading symbols from /usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/zfs.ko.debug...done. done. Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/zfs.ko Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/opensolaris.ko...Reading symbols from /usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/opensolaris.ko.debug... done. done. Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/opensolaris.ko Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/geom_eli.ko...Reading symbols from /usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/geom_eli.ko.debug...done. done. Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/geom_eli.ko Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/accf_http.ko...Reading symbols from /usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/accf_http.ko.debug...done. done. Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/accf_http.ko Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/coretemp.ko...Reading symbols from /usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/coretemp.ko.debug...done. done. Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/coretemp.ko Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/aesni.ko...Reading symbols from /usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/aesni.ko.debug...done. done. Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/aesni.ko Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/if_bridge.ko...Reading symbols from /usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/if_bridge.ko.debug...done. done. Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/if_bridge.ko Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/bridgestp.ko...Reading symbols from /usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/bridgestp.ko.debug...done. done. Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/bridgestp.ko Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/ums.ko...Reading symbols from /usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/ums.ko.debug...done. done. Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/ums.ko Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/netgraph.ko...Reading symbols from /usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/netgraph.ko.debug...done. done. Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/netgraph.ko Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/ng_netflow.ko...Reading symbols from /usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/ng_netflow.ko.debug...done. done. Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/ng_netflow.ko Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/ng_ksocket.ko...Reading symbols from /usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/ng_ksocket.ko.debug...done. done. Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/ng_ksocket.ko Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/ng_ether.ko...Reading symbols from /usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/ng_ether.ko.debug...done. done. Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/ng_ether.ko Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/ng_socket.ko...Reading symbols from /usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/ng_socket.ko.debug...done. done. Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/ng_socket.ko Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/linux.ko...Reading symbols from /usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/linux.ko.debug...done. done. Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/linux.ko Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/linux_common.ko...Reading symbols from /usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/linux_common.ko.debug...done. done. Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/linux_common.ko Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/linux64.ko...Reading symbols from /usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/linux64.ko.debug...done. done. Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/linux64.ko Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/fdescfs.ko...Reading symbols from /usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/fdescfs.ko.debug...done. done. Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/fdescfs.ko Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/if_epair.ko...Reading symbols from /usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/if_epair.ko.debug...done. done. Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/if_epair.ko Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/nullfs.ko...Reading symbols from /usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/nullfs.ko.debug...done. done. Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/nullfs.ko Reading symbols from /boot/kernel.GENERIC-NODEBUG-VIMAGE/tmpfs.ko...Reading symbols from /usr/lib/debug//boot/kernel.GENERIC-NODEBUG-VIMAGE/tmpfs.ko.debug...done. done. Loaded symbols for /boot/kernel.GENERIC-NODEBUG-VIMAGE/tmpfs.ko #0 doadump (textdump=1) at pcpu.h:221 221 __asm("movq %%gs:%1,%0" : "=r" (td) (kgdb) list *0xffffffff80b4d91c 0xffffffff80b4d91c is in turnstile_broadcast (/usr/src/sys/kern/subr_turnstile.c:837). 832 833 /* 834 * Transfer the blocked list to the pending list. 835 */ 836 mtx_lock_spin(&td_contested_lock); 837 TAILQ_CONCAT(&ts->ts_pending, &ts->ts_blocked[queue], td_lockq); 838 mtx_unlock_spin(&td_contested_lock); 839 840 /* 841 * Give a turnstile to each thread. The last thread gets Current language: auto; currently minimal (kgdb) backtrace #0 doadump (textdump=1) at pcpu.h:221 #1 0xffffffff80aea40e in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:366 #2 0xffffffff80aea9db in vpanic (fmt=<value optimized out>, ap=<value optimized out>) at /usr/src/sys/kern/kern_shutdown.c:759 #3 0xffffffff80aea813 in panic (fmt=0x0) at /usr/src/sys/kern/kern_shutdown.c:690 #4 0xffffffff8039e197 in db_panic (addr=<value optimized out>, have_addr=false, count=0, modif=0x0) at /usr/src/sys/ddb/db_command.c:486 #5 0xffffffff8039d689 in db_command (cmd_table=<value optimized out>) at /usr/src/sys/ddb/db_command.c:453 #6 0xffffffff8039d3e4 in db_command_loop () at /usr/src/sys/ddb/db_command.c:506 #7 0xffffffff803a053b in db_trap (type=<value optimized out>, code=<value optimized out>) at /usr/src/sys/ddb/db_main.c:251 #8 0xffffffff80b36b33 in kdb_trap (type=<value optimized out>, code=<value optimized out>, tf=<value optimized out>) at /usr/src/sys/kern/subr_kdb.c:654 #9 0xffffffff80fdd441 in trap_fatal (frame=0xfffffe046813a390, eva=48) at /usr/src/sys/amd64/amd64/trap.c:836 #10 0xffffffff80fdd673 in trap_pfault (frame=0xfffffe046813a390, usermode=0) at /usr/src/sys/amd64/amd64/trap.c:691 #11 0xffffffff80fdcbfc in trap (frame=0xfffffe046813a390) at /usr/src/sys/amd64/amd64/trap.c:442 #12 0xffffffff80fbf491 in calltrap () at /usr/src/sys/amd64/amd64/exception.S:236 #13 0xffffffff80b4d91c in turnstile_broadcast (ts=0x0, queue=1) at /usr/src/sys/kern/subr_turnstile.c:837 #14 0xffffffff80ae5e1f in __rw_wunlock_hard (c=0xfffff803f886d960, tid=<value optimized out>, file=<value optimized out>, line=<value optimized out>) at /usr/src/sys/kern/kern_rwlock.c:1027 #15 0xffffffff80e525dc in vm_map_delete (map=<value optimized out>, start=<value optimized out>, end=<value optimized out>) at /usr/src/sys/vm/vm_map.c:2960 #16 0xffffffff80e54477 in vm_map_remove (map=0xfffff8035540f000, start=140737488355328, end=1) at /usr/src/sys/vm/vm_map.c:3077 #17 0xffffffff80a9863f in exec_new_vmspace (imgp=0xfffffe046813a860, sv=0xffffffff81a596e8) at /usr/src/sys/kern/kern_exec.c:1096 #18 0xffffffff80a6ced8 in exec_elf64_imgact (imgp=<value optimized out>) at /usr/src/sys/kern/imgact_elf.c:896 #19 0xffffffff80a9670d in kern_execve (td=<value optimized out>, args=<value optimized out>, mac_p=0x0) at /usr/src/sys/kern/kern_exec.c:603 #20 0xffffffff80a95b9c in sys_execve (td=0xfffff8032893aa00, uap=0xfffffe046813ab80) at /usr/src/sys/kern/kern_exec.c:219 #21 0xffffffff80fddde8 in amd64_syscall (td=<value optimized out>, traced=0) at subr_syscall.c:135 #22 0xffffffff80fbf77b in Xfast_syscall () at /usr/src/sys/amd64/amd64/exception.S:396 #23 0x0000000800b468ea in ?? () Previous frame inner to this frame (corrupt stack?) (kgdb) up 11 #11 0xffffffff80fdcbfc in trap (frame=0xfffffe046813a390) at /usr/src/sys/amd64/amd64/trap.c:442 442 (void) trap_pfault(frame, FALSE); (kgdb) list 437 438 KASSERT(cold || td->td_ucred != NULL, 439 ("kernel trap doesn't have ucred")); 440 switch (type) { 441 case T_PAGEFLT: /* page fault */ 442 (void) trap_pfault(frame, FALSE); 443 goto out; 444 445 case T_DNA: 446 if (PCB_USER_FPU(td->td_pcb)) (kgdb) print td $1 = (struct thread *) 0xfffff8032893aa00 (kgdb) print td->td_ucred $2 = (struct ucred *) 0xfffff8004005ec00 (kgdb) print type $3 = 12 (kgdb) up #12 0xffffffff80fbf491 in calltrap () at /usr/src/sys/amd64/amd64/exception.S:236 236 call trap_check Current language: auto; currently asm (kgdb) up #13 0xffffffff80b4d91c in turnstile_broadcast (ts=0x0, queue=1) at /usr/src/sys/kern/subr_turnstile.c:837 837 TAILQ_CONCAT(&ts->ts_pending, &ts->ts_blocked[queue], td_lockq); Current language: auto; currently minimal (kgdb) up #14 0xffffffff80ae5e1f in __rw_wunlock_hard (c=0xfffff803f886d960, tid=<value optimized out>, file=<value optimized out>, line=<value optimized out>) at /usr/src/sys/kern/kern_rwlock.c:1027 1027 turnstile_broadcast(ts, queue); (kgdb) up #15 0xffffffff80e525dc in vm_map_delete (map=<value optimized out>, start=<value optimized out>, end=<value optimized out>) at /usr/src/sys/vm/vm_map.c:2960 2960 VM_OBJECT_WUNLOCK(object); (kgdb) up #16 0xffffffff80e54477 in vm_map_remove (map=0xfffff8035540f000, start=140737488355328, end=1) at /usr/src/sys/vm/vm_map.c:3077 3077 result = vm_map_delete(map, start, end); (kgdb) up #17 0xffffffff80a9863f in exec_new_vmspace (imgp=0xfffffe046813a860, sv=0xffffffff81a596e8) at /usr/src/sys/kern/kern_exec.c:1096 1096 vm_map_remove(map, vm_map_min(map), vm_map_max(map)); (kgdb) up #18 0xffffffff80a6ced8 in exec_elf64_imgact (imgp=<value optimized out>) at /usr/src/sys/kern/imgact_elf.c:896 896 error = exec_new_vmspace(imgp, sv); (kgdb) up #19 0xffffffff80a9670d in kern_execve (td=<value optimized out>, args=<value optimized out>, mac_p=0x0) at /usr/src/sys/kern/kern_exec.c:603 603 error = (*execsw[i]->ex_imgact)(imgp); (kgdb) up #20 0xffffffff80a95b9c in sys_execve (td=0xfffff8032893aa00, uap=0xfffffe046813ab80) at /usr/src/sys/kern/kern_exec.c:219 219 error = kern_execve(td, &args, NULL); (kgdb) up #21 0xffffffff80fddde8 in amd64_syscall (td=<value optimized out>, traced=0) at subr_syscall.c:135 135 error = (sa->callp->sy_call)(td, sa->args); (kgdb) up #22 0xffffffff80fbf77b in Xfast_syscall () at /usr/src/sys/amd64/amd64/exception.S:396 396 call amd64_syscall Current language: auto; currently asm (kgdb) up #23 0x0000000800b468ea in ?? () (kgdb) up Initial frame selected; you cannot go up. (kgdb) quit Regards, Ben -- From: Benjamin Woods woodsb02_at_gmail.comReceived on Thu Sep 22 2016 - 10:48:31 UTC
This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:41:08 UTC