Re: ZFS gets stuck

On Sun, Dec 07, 2008 at 11:21:13AM +0100, Stefan Bethke wrote:

  For the past week, I've been stress testing two new boxes by running  
  make -j4 universe. /usr/src is on ufs, /usr/obj on zfs, backed by a  
  single disk pool. Every so often (about once every one or two days),  
  processes start getting wedged on accessing the zfs file systems.

  FreeBSD lokschuppen.lassitu.de 8.0-CURRENT FreeBSD 8.0-CURRENT #1: Wed  
  Dec  3 07:05:03 UTC 2008     root_at_lokschuppen.lassitu.de:/usr/obj/usr/ 
  src/sys/EISENBOOT  amd64
  
  So far, I've had this in loader.conf:
  vfs.zfs.arc_max="512M"
  vfs.zfs.prefetch_disable="1"
  
  
  I'm now adding
  vfs.zfs.zil_disable="1"
  
  to see if that makes a difference.
  
  Is there anything in particular people would want me to check out?  
  Kernel is GENERIC minus a number of devices, and without INVARIANTS  
  and WITNESS.
  
I have one system running a recent -current with 2G of ram that would 
tend to get stuck almost nightly during a nightly rsync, and have been
trying to tune it to get rid of the problems.  I've made it about 6 days
so far using just:

vm.kmem_size=2G
vm.kmem_size_max=2G
vfs.zfs.arc_max=512M

With these settings, I had two rsyncs running constantly for probably 12 
hours on the first day or two and still once nightly, so far so good.  I 
tried leaving the kmem settings out and while vm.kmem_size_max appeared to 
auto tune to approx 4.5G, I still had a out of kmem panic (a low number 
around 600 or 900M, I don't remember) until I boosted both kmem settings to 
2G.  I previously had ~1.6G for the kmem settings from when that was the 
highest they could be set, and maybe that wasn't high enough with recent 
-current because it had hangs every few days as opposed to every few months 
with zfs "6" (same zfs as in -stable).  I might just be having a string of 
good luck, and the data that rsync transfers might have played a role (it 
changes a varying amount each day).  If it looks fairly stable then I will 
apply the settings to a different, busier server with ZFS.