Re: ZFS command can block the whole ZFS subsystem!

In the last episode (Jan 03), O. Hartmann said:
> On Fri, 3 Jan 2014 14:38:03 -0000 "Steven Hartland" <killing_at_multiplay.co.uk> wrote:
> > From: "O. Hartmann" <ohartman_at_zedat.fu-berlin.de>
> > > For some security reasons, I dumped via "dd" a large file onto a 3TB
> > > disk.  The systems is 11.0-CURRENT #1 r259667: Fri Dec 20 22:43:56 CET
> > > 2013 amd64.  Filesystem in question is a single ZFS pool.
> > > 
> > > Issuing the command
> > > 
> > > "rm dumpfile.txt"
> > > 
> > > and then hitting Ctrl-Z to bring the rm command into background via
> > > fg" (I use FreeBSD's csh in that console) locks up the entire command
> > > and even worse - it seems to wind up the pool in question for being
> > > exported!
> >
> > You can check that gstat -d
> 
> command report 100% acticity on the drive. I exported the pool in
> question in single user mode and now try to import it back while in
> miltiuser mode.

Did you happen to have enabled deduplication on the filesystem in question? 
That's the only thing I can think of that would make file deletions run
slow.  I have deleted files up to 10GB on regular filesystems with no
noticable delay at the commandline.  If you have deduplication enabled,
however, each block's hash has to be looked up in the dedupe table, and if
you don't have enough RAM for it to be loaded completely into memory, that
will be very very slow :)

There are varying recommendations on how much RAM you need for a given pool
size, since the DDT has to hold an entry for each block written, and
blocksize depends on whether you wrote your files sequentially (128K blocks)
or randomly (8k or smaller).  Each DDT entry takes 320 bytes of RAM, so a
full 3TB ZFS pool would need at minimum 320*(3TB/128K) ~= 7GB of RAM to hold
the DDT, and much more than that if your averge blocksize is less than 128K.

So, if your system has less than 8GB of RAM in it, there's no way the DDT
will be able to stay in memory, so you're probably going to have to do at
least one disk seek (probably more, since you're writing to the DDT as well)
per block in the file you're deleting.  You should probably have 16GB or
more RAM, and use an SSD as a L2ARC device as well.

-- 
	Dan Nelson
	dnelson_at_allantgroup.com