Re: My ZFS v28 Testing Experience

On Wed, Jan 12, 2011 at 11:03:19PM -0400, Chris Forgeron wrote:
> I've been testing out the v28 patch code for a month now, and I've yet to report any real issues other than what is mentioned below. 
> 
> I'll detail some of the things I've tested, hopefully the stability of v28 in FreeBSD will convince others to give it a try so the final release of v28 will be as solid as possible.
> 
> I've been using FreeBSD 9.0-CURRENT as of Dec 12th, and 8.2PRE as of Dec 16th
> 
> What's worked well:
> 
> - I've made and destroyed small raidz's (3-5 disks), large 26 disk raid-10's, and a large 20 disk raid-50.
> - I've upgraded from v15, zfs 4, no issues on the different arrays noted above
> - I've confirmed that a v15 or v28 pool will import into Solaris 11 Express, and vice versa, with the exception about dual log or cache devices noted below. 
> - I've run many TB of data through the ZFS storage via benchmarks from my VM's connected via NFS, to simple copies inside the same pool, or copies from one pool to another. 
> - I've tested pretty much every compression level, and changing them as I tweak my setup and try to find the best blend.
> - I've added and subtracted many a log and cache device, some in failed states from hot-removals, and the pools always stayed intact.

Thank you very much for all your testing, that's really a valuable
contribution. I'll be happy to work with you on tracking down the
bottleneck in ZFSv28.

> Issues:
> 
> - Import of pools with multiple cache or log devices. (May be a very minor point)
> 
> A v28 pool created in Solaris 11 Express with 2 or more log devices, or 2 or more cache devices won't import in FreeBSD 9. This also applies to a pool that is created in FreeBSD, is imported in Solaris to have the 2 log devices added there, then exported and attempted to be imported back in FreeBSD. No errors, zpool import just hangs forever. If I reboot into Solaris, import the pool, remove the dual devices, then reboot into FreeBSD, I can then import the pool without issue. A single cache, or log device will import just fine. Unfortunately I deleted my witness-enabled FreeBSD-9 drive, so I can't easily fire it back up to give more debug info. I'm hoping some kind soul will attempt this type of transaction and report more detail to the list.
> 
> Note - I just decided to try adding 2 cache devices to a raidz pool in FreeBSD, export, and then importing, all without rebooting. That seems to work. BUT - As soon as you try to reboot FreeBSD with this pool staying active, it hangs on boot. Booting into Solaris, removing the 2 cache devices, then booting back into FreeBSD then works. Something is kept in memory between exporting then importing that allows this to work.  

Unfortunately I'm unable to reproduce this. It works for me with 2 cache
and 2 log vdevs. I tried to reboot, etc. My test exactly looks like
this:

	# zpool create tank raidz ada0 ada1
	# zpool add tank cache ada0 ada1
	# zpool export tank
	# kldunload zfs
	# zpool import tank
	<works>
	# reboot
	<works>

> - Speed. (More of an issue, but what do we do?)
> 
> Wow, it's much slower than Solaris 11 Express for transactions. I do understand that Solaris will have a slight advantage over any port of ZFS. All of my speed tests are made with a kernel without debug, and yes, these are -CURRENT and -PRE releases, but the speed difference is very large.

Before we go any further could you please confirm that you commented out
this line in sys/modules/zfs/Makefile:

	CFLAGS+=-DDEBUG=1

This turns all kind of ZFS debugging and slows it down a lot, but for
the correctness testing is invaluable. This will be turned off once we
import ZFS into FreeBSD-CURRENT.

BTW. In my testing Solaris 11 Express is much, much slower than
FreeBSD/ZFSv28. And by much I mean two or more times in some tests.
I was wondering if they have some debug turned on in Express.

> At first, I thought it may be more of an issue with the ix0/Intel X520DA2 10Gbe drivers that I'm using, since the bulk of my tests are over NFS (I'm going to use this as a SAN via NFS, so I test in that environment). 
> 
> But - I did a raw cp command from one pool to another of several TB. I executed the same command under FreeBSD as I did under Solaris 11 Express. When executed in FreeBSD, the copy took 36 hours. With a fresh destination pool of the same settings/compression/etc under Solaris, the copy took 7.5 hours. 

When you turn off compression (because it turns all-zero blocks into
holes) you can test it by simply:

	# dd if=/dev/zero of=/<zfs_fs>/zero bs=1m

-- 
Pawel Jakub Dawidek                       http://www.wheelsystems.com
pjd_at_FreeBSD.org                           http://www.FreeBSD.org
FreeBSD committer                         Am I Evil? Yes, I Am!