Oops, yea I see that now. In the krpc it is very hard to tell when the data structures (that hold the mutexes) can be safely free'd. Code like xprt_unregister() get called asynchronously and lock a mutex as soon as called. (The crash fixed by r313735 was a prematurely free'd mutex that xprt_unregister() used, but on the server side.) I'll look at the code as see if I can figure out how to delay freeing the structure without leaving it allocated "forever". (I'll admit I've been tempted to just never free it, since the memory leak this would cause would be small enough nothing would really notice it. And, of course for your case of shutdown, it would be harmless to just not free it.) rick ________________________________________ From: Larry Rosenman <ler_at_lerctr.org> Sent: Thursday, March 16, 2017 7:46:51 PM To: Rick Macklem; freebsd-fs_at_freebsd.org Cc: freebsd-current_at_FreeBSD.org Subject: Re: crash: umount_nfs: Current Err, I’m at r315289…. -- Larry Rosenman http://www.lerctr.org/~ler Phone: +1 214-642-9640 E-Mail: ler_at_lerctr.org US Mail: 17716 Limpia Crk, Round Rock, TX 78664-7281 On 3/16/17, 5:51 PM, "Rick Macklem" <rmacklem_at_uoguelph.ca> wrote: I believe the cause of this crash was fixed by a recent commit to head r313735 (which was MFC'd to stable/11 and stable/10). rick ________________________________________ From: owner-freebsd-current_at_freebsd.org <owner-freebsd-current_at_freebsd.org> on behalf of Larry Rosenman <ler_at_lerctr.org> Sent: Wednesday, March 15, 2017 10:44:33 PM To: freebsd-fs_at_freebsd.org Cc: freebsd-current_at_FreeBSD.org Subject: crash: umount_nfs: Current Recent current, playing with new FreeNAS Corral, client is FreeBSD -CURRENT. Lovely crash: borg.lerctr.org dumped core - see /var/crash/vmcore.1 Wed Mar 15 21:38:53 CDT 2017 FreeBSD borg.lerctr.org 12.0-CURRENT FreeBSD 12.0-CURRENT #11 r315289: Tue Mar 14 20:55:36 CDT 2017 root_at_borg.lerctr.org:/usr/obj/usr/src/sys/VT-LER amd64 panic: general protection fault GNU gdb (GDB) 7.12.1 [GDB v7.12.1 for FreeBSD] Copyright (C) 2017 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-portbld-freebsd12.0". Type "show configuration" for configuration details. For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>. For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from /boot/kernel/kernel...Reading symbols from /usr/lib/debug//boot/kernel/kernel.debug...done. done. Unread portion of the kernel message buffer: Fatal trap 9: general protection fault while in kernel mode cpuid = 1; apic id = 21 instruction pointer = 0x20:0xffffffff80a327ae stack pointer = 0x28:0xfffffe535ebb2800 frame pointer = 0x28:0xfffffe535ebb2830 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 3172 (umount) trap number = 9 panic: general protection fault cpuid = 1 time = 1489631515 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe535ebb2440 vpanic() at vpanic+0x19c/frame 0xfffffe535ebb24c0 panic() at panic+0x43/frame 0xfffffe535ebb2520 trap_fatal() at trap_fatal+0x322/frame 0xfffffe535ebb2570 trap() at trap+0x5e/frame 0xfffffe535ebb2730 calltrap() at calltrap+0x8/frame 0xfffffe535ebb2730 --- trap 0x9, rip = 0xffffffff80a327ae, rsp = 0xfffffe535ebb2800, rbp = 0xfffffe535ebb2830 --- __mtx_lock_flags() at __mtx_lock_flags+0x3e/frame 0xfffffe535ebb2830 xprt_unregister() at xprt_unregister+0x28/frame 0xfffffe535ebb2850 clnt_reconnect_destroy() at clnt_reconnect_destroy+0x38/frame 0xfffffe535ebb2880 nfs_unmount() at nfs_unmount+0x182/frame 0xfffffe535ebb28d0 dounmount() at dounmount+0x5c1/frame 0xfffffe535ebb2950 sys_unmount() at sys_unmount+0x30f/frame 0xfffffe535ebb2a70 amd64_syscall() at amd64_syscall+0x55a/frame 0xfffffe535ebb2bf0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe535ebb2bf0 --- syscall (22, FreeBSD ELF64, sys_unmount), rip = 0x800872b9a, rsp = 0x7fffffffde88, rbp = 0x7fffffffe3c0 --- Uptime: 2h43m8s Dumping 5744 out of 131005 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91% Reading symbols from /boot/kernel/zfs.ko...Reading symbols from /usr/lib/debug//boot/kernel/zfs.ko.debug...done. done. Reading symbols from /boot/kernel/opensolaris.ko...Reading symbols from /usr/lib/debug//boot/kernel/opensolaris.ko.debug...done. done. Reading symbols from /boot/kernel/linux.ko...Reading symbols from /usr/lib/debug//boot/kernel/linux.ko.debug...done. done. Reading symbols from /boot/kernel/linux_common.ko...Reading symbols from /usr/lib/debug//boot/kernel/linux_common.ko.debug...done. done. Reading symbols from /boot/kernel/if_lagg.ko...Reading symbols from /usr/lib/debug//boot/kernel/if_lagg.ko.debug...done. done. Reading symbols from /boot/kernel/coretemp.ko...Reading symbols from /usr/lib/debug//boot/kernel/coretemp.ko.debug...done. done. Reading symbols from /boot/kernel/aesni.ko...Reading symbols from /usr/lib/debug//boot/kernel/aesni.ko.debug...done. done. Reading symbols from /boot/kernel/filemon.ko...Reading symbols from /usr/lib/debug//boot/kernel/filemon.ko.debug...done. done. Reading symbols from /boot/kernel/fuse.ko...Reading symbols from /usr/lib/debug//boot/kernel/fuse.ko.debug...done. done. Reading symbols from /boot/kernel/ichsmb.ko...Reading symbols from /usr/lib/debug//boot/kernel/ichsmb.ko.debug...done. done. Reading symbols from /boot/kernel/smbus.ko...Reading symbols from /usr/lib/debug//boot/kernel/smbus.ko.debug...done. done. Reading symbols from /boot/kernel/ichwd.ko...Reading symbols from /usr/lib/debug//boot/kernel/ichwd.ko.debug...done. done. Reading symbols from /boot/kernel/cpuctl.ko...Reading symbols from /usr/lib/debug//boot/kernel/cpuctl.ko.debug...done. done. Reading symbols from /boot/kernel/cryptodev.ko...Reading symbols from /usr/lib/debug//boot/kernel/cryptodev.ko.debug...done. done. Reading symbols from /boot/kernel/dtraceall.ko...Reading symbols from /usr/lib/debug//boot/kernel/dtraceall.ko.debug...done. done. Reading symbols from /boot/kernel/profile.ko...Reading symbols from /usr/lib/debug//boot/kernel/profile.ko.debug...done. done. Reading symbols from /boot/kernel/dtrace.ko...Reading symbols from /usr/lib/debug//boot/kernel/dtrace.ko.debug...done. done. Reading symbols from /boot/kernel/systrace_freebsd32.ko...Reading symbols from /usr/lib/debug//boot/kernel/systrace_freebsd32.ko.debug...done. done. Reading symbols from /boot/kernel/systrace.ko...Reading symbols from /usr/lib/debug//boot/kernel/systrace.ko.debug...done. done. Reading symbols from /boot/kernel/sdt.ko...Reading symbols from /usr/lib/debug//boot/kernel/sdt.ko.debug...done. done. Reading symbols from /boot/kernel/fasttrap.ko...Reading symbols from /usr/lib/debug//boot/kernel/fasttrap.ko.debug...done. done. Reading symbols from /boot/kernel/fbt.ko...Reading symbols from /usr/lib/debug//boot/kernel/fbt.ko.debug...done. done. Reading symbols from /boot/kernel/dtnfscl.ko...Reading symbols from /usr/lib/debug//boot/kernel/dtnfscl.ko.debug...done. done. Reading symbols from /boot/kernel/dtmalloc.ko...Reading symbols from /usr/lib/debug//boot/kernel/dtmalloc.ko.debug...done. done. Reading symbols from /boot/modules/vboxdrv.ko...(no debugging symbols found)...done. Reading symbols from /boot/kernel/ipmi.ko...Reading symbols from /usr/lib/debug//boot/kernel/ipmi.ko.debug...done. done. Reading symbols from /boot/kernel/ipmi_linux.ko...Reading symbols from /usr/lib/debug//boot/kernel/ipmi_linux.ko.debug...done. done. Reading symbols from /boot/kernel/hwpmc.ko...Reading symbols from /usr/lib/debug//boot/kernel/hwpmc.ko.debug...done. done. Reading symbols from /boot/kernel/mfip.ko...Reading symbols from /usr/lib/debug//boot/kernel/mfip.ko.debug...done. done. Reading symbols from /boot/kernel/ums.ko...Reading symbols from /usr/lib/debug//boot/kernel/ums.ko.debug...done. done. Reading symbols from /boot/modules/vboxnetflt.ko...(no debugging symbols found)...done. Reading symbols from /boot/kernel/netgraph.ko...Reading symbols from /usr/lib/debug//boot/kernel/netgraph.ko.debug...done. done. Reading symbols from /boot/kernel/ng_ether.ko...Reading symbols from /usr/lib/debug//boot/kernel/ng_ether.ko.debug...done. done. Reading symbols from /boot/modules/vboxnetadp.ko...(no debugging symbols found)...done. Reading symbols from /boot/kernel/linux64.ko...Reading symbols from /usr/lib/debug//boot/kernel/linux64.ko.debug...done. done. Reading symbols from /boot/kernel/linprocfs.ko...Reading symbols from /usr/lib/debug//boot/kernel/linprocfs.ko.debug...done. done. Reading symbols from /boot/kernel/tmpfs.ko...Reading symbols from /usr/lib/debug//boot/kernel/tmpfs.ko.debug...done. done. __curthread () at ./machine/pcpu.h:232 232 __asm("movq %%gs:%1,%0" : "=r" (td) (kgdb) #0 __curthread () at ./machine/pcpu.h:232 #1 doadump (textdump=1) at /usr/src/sys/kern/kern_shutdown.c:318 #2 0xffffffff80a52c15 in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:386 #3 0xffffffff80a53206 in vpanic (fmt=<optimized out>, ap=0xfffffe535ebb2500) at /usr/src/sys/kern/kern_shutdown.c:779 #4 0xffffffff80a53253 in panic (fmt=<unavailable>) at /usr/src/sys/kern/kern_shutdown.c:710 #5 0xffffffff80ecd5c2 in trap_fatal (frame=0xfffffe535ebb2740, eva=0) at /usr/src/sys/amd64/amd64/trap.c:801 #6 0xffffffff80eccb8e in trap (frame=0xfffffe535ebb2740) at /usr/src/sys/amd64/amd64/trap.c:197 #7 <signal handler called> #8 __mtx_lock_flags (c=0xdeadc0dedeadc0f6, opts=0, file=0xffffffff8147e721 "/usr/src/sys/rpc/svc.c", line=380) at /usr/src/sys/kern/kern_mutex.c:239 #9 0xffffffff80cc14b8 in xprt_unregister (xprt=0xfffff8022e0fce00) at /usr/src/sys/rpc/svc.c:380 #10 0xffffffff80cbc058 in clnt_reconnect_destroy (cl=0xfffff80146fa3900) at /usr/src/sys/rpc/clnt_rc.c:500 #11 0xffffffff80969972 in nfs_unmount (mp=<optimized out>, mntflags=<optimized out>) at /usr/src/sys/fs/nfsclient/nfs_clvfsops.c:1704 #12 0xffffffff80b0b761 in dounmount (mp=0xdeadc0dedeadc0f6, flags=134217728, td=0xfffff8018f2eb000) at /usr/src/sys/kern/vfs_mount.c:1388 #13 0xffffffff80b0b11f in sys_unmount (td=0xfffff8018f2eb000, uap=0xfffffe535ebb2b70) at /usr/src/sys/kern/vfs_mount.c:1215 #14 0xffffffff80ecdfea in syscallenter (td=0xfffff8018f2eb000, sa=<optimized out>) at /usr/src/sys/amd64/amd64/../../kern/subr_syscall.c:135 #15 amd64_syscall (td=0xfffff8018f2eb000, traced=0) at /usr/src/sys/amd64/amd64/trap.c:902 #16 <signal handler called> Can't read data for section '.eh_frame' in file '/' (kgdb) vmcore / source IS available. -- Larry Rosenman http://www.lerctr.org/~ler Phone: +1 214-642-9640 E-Mail: ler_at_lerctr.org US Mail: 17716 Limpia Crk, Round Rock, TX 78664-7281 _______________________________________________ freebsd-current_at_freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscribe_at_freebsd.org"Received on Fri Mar 17 2017 - 20:31:47 UTC
This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:41:10 UTC