Re: ZFS on 32-bit CPUs?

On Wed, Jun 06, 2007 at 02:19:57AM +0200, Ivan Voras wrote:
> Sean Hafeez wrote:
> > Has anyone looked at the ZFS port and how it does on 32-bit CPUs vs
> > 64-bit ones? I know under Solaris they do not recommend using a 32-bit
> > CPU. I my case I was thinking about doing some testing on a Dual P3-850.
> 
> It works, and there's never been doubt that it would work. The main
> resource you need is memory. At least 1 GB is recommended, but it should
> work with 512 MB (though people were reporting panics unless they scale
> ZFS and VFS parameters down). If you're thinking of using it in
> production, you should read the threads on this list regarding ZFS,
> especially those mentioning panics.

It "works", but there are serious performance issues to do with how
ZFS on freebsd handles caching of data.  In order to get reasonable
performance you will want to tune VM_KMEM_SIZE_MAX as high as you can
get away with (depends on how much ram you have).  Roughly half of
this will be used by the ARC (zfs buffer cache).  This is typically
less memory than the standard buffer cache would have available so ZFS
still loses out on caching, particularly on systems with a lot of RAM.

You may also need to hack zfs a bit.  The following patch improves
performance for me on amd64 (and avoids a deadlock).  I have not
tested whether it is sufficient or reasonable on i386 (only amd64),
the KVA shortage there makes it hard to tune memory availability the
way zfs wants it.

There is also a panic condition that may be triggered on SMP when you
have INVARIANTS enabled.  pjd and I don't yet understand the cause of
this but it appears to be spurious ("returning to userspace with 1
locks held" when no locks appear to actually be held, i.e. it seems to
be some kind of leak in the stats).

Index: contrib/opensolaris/uts/common/fs/zfs/arc.c
===================================================================
RCS file: /mnt/xor/ncvs/src/sys/contrib/opensolaris/uts/common/fs/zfs/arc.c,v
retrieving revision 1.9
diff -u -d -u -r1.9 arc.c
--- contrib/opensolaris/uts/common/fs/zfs/arc.c	23 Apr 2007 21:52:14 -0000	1.9
+++ contrib/opensolaris/uts/common/fs/zfs/arc.c	2 Jun 2007 20:22:00 -0000
_at__at_ -1439,8 +1439,10 _at__at_
 		return (1);
 #endif
 #else
-	if (kmem_used() > kmem_size() / 2)
+	if (kmem_used() * 10 > kmem_size() * 9) {
+		printf("kmem_used = %ld, kmem_size = %ld\n", kmem_used(), kmem_size());
 		return (1);
+	}
 #endif
 
 #else
_at__at_ -2689,13 +2689,19 _at__at_
 static void
 arc_lowmem(void *arg __unused, int howto __unused)
 {
+	int vnodesave, count = 0;
 
 	/* Serialize access via arc_lowmem_lock. */
 	mutex_enter(&arc_lowmem_lock);
 	zfs_needfree = 1;
 	cv_signal(&arc_reclaim_thr_cv);
-	while (zfs_needfree)
+	vnodesave = desiredvnodes;
+	while (zfs_needfree) {
+		if (count++ % 5 == 0)
+			desiredvnodes /= 2;
 		tsleep(&zfs_needfree, 0, "zfs:lowmem", hz / 5);
+	}
+	desiredvnodes = vnodesave;
 	mutex_exit(&arc_lowmem_lock);
 }
 #endif