Use after Free panic: ZFS?

I've seen a couple of these...
⌂70% [ler_at_borg.lerctr.org:/var/crash] $ uname -aKU
FreeBSD borg.lerctr.org 13.0-CURRENT FreeBSD 13.0-CURRENT r343437 
LER-MINIMAL  amd64 1300009 1300009
⌂66% [ler_at_borg.lerctr.org:/var/crash] $

Ideas?  vmcore/symbols available.

borg.lerctr.org dumped core - see /var/crash/vmcore.7

Tue Jan 29 04:00:46 CST 2019

FreeBSD borg.lerctr.org 13.0-CURRENT FreeBSD 13.0-CURRENT r343437 
LER-MINIMAL  amd64

panic: Memory modified after free 0xfffff807019ca980(32) val=0 _at_ 
0xfffff807019ca980

GNU gdb (GDB) 8.2 [GDB v8.2 for FreeBSD]
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 
<http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-portbld-freebsd13.0".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
     <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /boot/kernel/kernel...Reading symbols from 
/usr/lib/debug//boot/kernel/kernel.debug...done.
done.

Unread portion of the kernel message buffer:
panic: Memory modified after free 0xfffff807019ca980(32) val=0 _at_ 
0xfffff807019ca980

cpuid = 5
time = 1548755136
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 
0xfffffe00f750c880
vpanic() at vpanic+0x1b4/frame 0xfffffe00f750c8e0
panic() at panic+0x43/frame 0xfffffe00f750c940
trash_ctor() at trash_ctor+0x4c/frame 0xfffffe00f750c950
uma_zalloc_arg() at uma_zalloc_arg+0x9df/frame 0xfffffe00f750c9e0
uma_zfree_arg() at uma_zfree_arg+0x46a/frame 0xfffffe00f750ca40
arc_buf_destroy_impl() at arc_buf_destroy_impl+0x133/frame 
0xfffffe00f750ca80
arc_buf_destroy() at arc_buf_destroy+0x17a/frame 0xfffffe00f750cab0
dbuf_destroy() at dbuf_destroy+0x87/frame 0xfffffe00f750cb10
dbuf_evict_one() at dbuf_evict_one+0x187/frame 0xfffffe00f750cb40
dbuf_evict_thread() at dbuf_evict_thread+0x185/frame 0xfffffe00f750cbb0
fork_exit() at fork_exit+0x84/frame 0xfffffe00f750cbf0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00f750cbf0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
Uptime: 3d16h49m14s
Dumping 22587 out of 131028 
MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91%

__curthread () at ./machine/pcpu.h:230
230             __asm("movq %%gs:%P1,%0" : "=r" (td) : "n" 
(OFFSETOF_CURTHREAD));
(kgdb) #0  __curthread () at ./machine/pcpu.h:230
#1  doadump (textdump=<optimized out>)
     at /usr/src/sys/kern/kern_shutdown.c:371
#2  0xffffffff80491760 in kern_reboot (howto=260)
     at /usr/src/sys/kern/kern_shutdown.c:451
#3  0xffffffff80491bc0 in vpanic (fmt=<optimized out>, 
ap=0xfffffe00f750c920)
     at /usr/src/sys/kern/kern_shutdown.c:877
#4  0xffffffff80491913 in panic (fmt=<unavailable>)
     at /usr/src/sys/kern/kern_shutdown.c:804
#5  0xffffffff8071255c in trash_ctor (mem=<unavailable>, 
size=<unavailable>,
     arg=<optimized out>, flags=<optimized out>)
     at /usr/src/sys/vm/uma_dbg.c:82
#6  0xffffffff8070cf4f in uma_zalloc_arg (zone=0xfffff8203ffdc000,
     udata=0x108, flags=1) at /usr/src/sys/vm/uma_core.c:2418
#7  0xffffffff8070d69a in bucket_alloc (zone=<optimized out>,
     udata=<unavailable>, flags=<unavailable>)
     at /usr/src/sys/vm/uma_core.c:433
#8  uma_zfree_arg (zone=0xfffff801059a0000, item=<optimized out>,
     udata=0xfffff81042431940) at /usr/src/sys/vm/uma_core.c:3153
#9  0xffffffff812f8c13 in arc_free_data_buf (hdr=<optimized out>,
     buf=0xfffffe025fe1e000, size=8192, tag=<optimized out>)
     at 
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c:5248
#10 arc_buf_destroy_impl (buf=0xfffff8190202ef00)
     at 
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c:3270
#11 0xffffffff812f859a in arc_buf_destroy (buf=0xfffff8190202ef00,
     tag=0xfffff80aea618840)
     at 
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c:3687
#12 0xffffffff8130d3d7 in dbuf_destroy (db=0xfffff80aea618840)
     at 
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dbuf.c:2328
#13 0xffffffff81313bb7 in dbuf_evict_one ()
     at 
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dbuf.c:717
#14 0xffffffff8130b1d5 in dbuf_evict_thread (unused=<optimized out>)
     at 
/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dbuf.c:757
#15 0xffffffff80458a94 in fork_exit (
     callout=0xffffffff8130b050 <dbuf_evict_thread>, arg=0x0,
     frame=0xfffffe00f750cc00) at /usr/src/sys/kern/kern_fork.c:1055
#16 <signal handler called>
(kgdb)



-- 
Larry Rosenman                     http://www.lerctr.org/~ler
Phone: +1 214-642-9640                 E-Mail: ler_at_lerctr.org
US Mail: 5708 Sabbia Dr, Round Rock, TX 78665-2106