[ZFS] still kmem_too_small on recent current

Hi!

Just got a kmem_too_small panic on yesterday's current after writing
30GB disk image via NFS.

This is amd64 system, with 8GB mem and those tunables:

vm.kmem_size_max="2G"
vm.kmem_size="2G"
vfs.zfs.arc_max="1G"

ZFS is 6x1TB raidz2.

I have a coredump of it, so just ask if you need additional details.

---
#0  doadump () at pcpu.h:223
#1  0xffffffff8054bec6 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:420
#2  0xffffffff8054c356 in panic (fmt=Variable "fmt" is not available.
) at /usr/src/sys/kern/kern_shutdown.c:576
#3  0xffffffff806ed86e in kmem_malloc (map=0xffffff00010000e0, size=131072, flags=2) at /usr/src/sys/vm/vm_kern.c:304
#4  0xffffffff806e5e75 in uma_large_malloc (size=131072, wait=2) at /usr/src/sys/vm/uma_core.c:3001
#5  0xffffffff8053b271 in malloc (size=131072, mtp=0xffffffff80e1a1e0, flags=2) at /usr/src/sys/kern/kern_malloc.c:391
#6  0xffffffff80d90ca6 in vdev_queue_io_to_issue (vq=0xffffff00065bf420, pending_limit=Variable "pending_limit" is not available.
) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:227
#7  0xffffffff80d90e1c in vdev_queue_io_done (zio=0xffffff018e1995a0) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:313
#8  0xffffffff80da2370 in zio_vdev_io_done (zio=0xffffff013bc26870) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1845
#9  0xffffffff80da0a10 in zio_execute (zio=0xffffff013bc26870) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:996
#10 0xffffffff80d49f1c in taskq_thread (arg=Variable "arg" is not available.
) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/os/taskq.c:854
#11 0xffffffff80525fad in fork_exit (callout=0xffffffff80d49d48 <taskq_thread>, arg=0xffffff000188fd80, frame=0xffffff80c516cc90) at /usr/src/sys/kern/kern_fork.c:829
#12 0xffffffff8076d56e in fork_trampoline () at /usr/src/sys/amd64/amd64/exception.S:552
---

I've monitored kstat.zfs.misc.arcstats.size, top and vmstat during
it, and here are the observations:

- normal copying process uses ~1GB arc (sysctl) and ~1GB wired (top),
  which I assume includes arc.
- actual disk writes are not continuous, ZFS writes data in >512MB
  packs.
- usually those writes are pretty uniform (a write in ~10 seconds,
  while thoughput is 50MB/s, array can handle 400MB/s writes), but
  sometimes the write is either larger, or the disks can't keep up,
  or both - those moments are visible as much higher CPU load for
  3-5 seconds, wired kicks up to 1700 mb and more, and arc decreases
  to ~800MB. I assume the latter is recently committed VM backpressure,
  and the former means panic if it rolls over kmem_max.

Seems so, as I've just got another panic. Just curious, what uses that
much memory in those moments?

Here are last contents of ssh consoles:

---
kstat.zfs.misc.arcstats.size: 804244704
kstat.zfs.misc.arcstats.size: 804244704
kstat.zfs.misc.arcstats.size: 804244704
---
 procs      memory      page                    disks     faults         cpu
 r b w     avm    fre   flt  re  pi  po    fr  sr ad8 ad10   in   sy   cs us sy id
 0 0 0   1017M  5348M   308   0   0   0   253   0   0   0   12  411  655  0  0 100
 0 0 0   1017M  5348M   241   0   0   0   257   0   0   0   18  381  748  0  0 100
 0 0 0   1017M  5348M   308   0   0   0   253   0   0   0   16  411  645  0  0 100
---
last pid:  7620;  load averages:  1.91,  1.80,  1.42       up 0+00:48:56  05:44:11
83 processes:  1 running, 82 sleeping
CPU:  0.2% user,  0.0% nice,  0.0% system,  0.0% interrupt, 99.8% idle
Mem: 149M Active, 60M Inact, 2317M Wired, 1292K Cache, 821M Buf, 5347M Free
Swap: 10G Total, 10G Free
---

Will give it another go with 512MB arc and 4GB kmem.

-- 
Dmitry Marakasov   .   55B5 0596 FF1E 8D84 5F56  9510 D35A 80DD F9D2 F77D
amdmi3_at_amdmi3.ru  ..:  jabber: amdmi3_at_jabber.ru    http://www.amdmi3.ru