Re: ZFS RAIDZ1: resilvering at <17.3M/s => abyssal slow ...

Am Thu, 14 Dec 2017 14:09:39 +0100
Daniel Nebdal <dnebdal_at_gmail.com> schrieb:

> On Thu, Dec 14, 2017 at 12:48 PM, O. Hartmann <ohartmann_at_walstatt.org> wrote:
> > Hello out there,
> >
> > I had to replace a HDD on a RAIDZ1 pool comprised from 4 HDDs, each 3 TB (netto size
> > of the pool is about 8-9 TB, the used amount is reported to be ~ 6TB).
> >
> > I just started the rebuild/resilvering process and watch the pool crwaling at ~ 18
> > MB/s. At the moment, there is no load on the array, the host is a IvyBridge XEON with
> > 4 core/8 threads and 3,4 GHz and 16 GB of RAM. The HDDs are attached to a on-board
> > SATA II (300 MB/s max) Intel chip - this just for the record.
> >
> > Recently, I switch on the "sync" attribute on most of the defined pools's zfs
> > filesystems
> > - I also use a SSD for ZIL/L2ARC caching, but it seems to be unused recently in
> > FreeBSD CURRENT's ZFS - this from a observers perspective only.
> >
> > When scrubbing, I see recently also reduced performance on the pool, so I'm wondering
> > about the low throughput at the very moment when resilvering is in progress.
> >
> > If the "perspective" of "zpool status" is correct, then I have to wait after two hours
> > for another 100 hours - ~ 4 days? Ups ... I think there is something badly
> > misconfigured or missing.
> >
> > The pool is not "tuned" in any very sophisticated way, since I trust the kernels
> > ability to scale automatically - if I'm not wrong for amd64 ...
> >
> > Are there any aspects to look at?
> >
> > Thanks in advance,
> >
> > Oliver
> >
> > --
> > O. Hartmann
> >
> > Ich widerspreche der Nutzung oder Übermittlung meiner Daten für
> > Werbezwecke oder für die Markt- oder Meinungsforschung (§ 28 Abs. 4 BDSG).  
> 
> 
> This is kind of to be expected - for whatever reason, resilvers seem
> to go super slow at first and then speed up significantly. Just don't
> ask me how long "at first" is - I'd give it several (more) hours.
> I don't _think_ sync affects resilver speed (since that's a filesystem
> level setting and the resilver happens at the block level), but that's
> pure speculation.
> The Tuning guide at https://wiki.freebsd.org/ZFSTuningGuide also
> suggests a list of sysctls you can tweak, which could be worth a try:
> 
> ### wiki
> If you're getting horrible performance during a scrub or resilver, the
> following sysctls can be set:
> 
> vfs.zfs.scrub_delay=0
> vfs.zfs.top_maxinflight=128
> vfs.zfs.resilver_min_time_ms=5000
> vfs.zfs.resilver_delay=0
> 
> Setting those sysctls to those values increased my (Shawn Webb's)
> resilver performance from 7MB/s to 230MB/s.
> ###
> 
> 
> 

I had already set those tuning parameters, shortly after I sent the mail to the list, but
there wasn't any sign of increasing performance since then.

Now, after roughly 4 hours of running resilvering, it seems that the speed/throughput is
increasing: from initially 12 - 17 MB/s its now at ~ 90 MBytes/s - sounds more promising.
I remember that I whitnessed this also when I checked the last time scrubbing: a non
linear gradient in throughput while time shifts forward ...

So, everything seems to be all right ...

Kind regards,
Oliver

-- 
O. Hartmann

Ich widerspreche der Nutzung oder Übermittlung meiner Daten für
Werbezwecke oder für die Markt- oder Meinungsforschung (§ 28 Abs. 4 BDSG).