Re: ZFS RAIDZ1: resilvering at <17.3M/s => abyssal slow ...

On Thu, Dec 14, 2017 at 12:48 PM, O. Hartmann <ohartmann_at_walstatt.org> wrote:
> Hello out there,
>
> I had to replace a HDD on a RAIDZ1 pool comprised from 4 HDDs, each 3 TB (netto size of
> the pool is about 8-9 TB, the used amount is reported to be ~ 6TB).
>
> I just started the rebuild/resilvering process and watch the pool crwaling at ~ 18 MB/s.
> At the moment, there is no load on the array, the host is a IvyBridge XEON with 4 core/8
> threads and 3,4 GHz and 16 GB of RAM. The HDDs are attached to a on-board SATA II (300
> MB/s max) Intel chip - this just for the record.
>
> Recently, I switch on the "sync" attribute on most of the defined pools's zfs filesystems
> - I also use a SSD for ZIL/L2ARC caching, but it seems to be unused recently in FreeBSD
> CURRENT's ZFS - this from a observers perspective only.
>
> When scrubbing, I see recently also reduced performance on the pool, so I'm wondering
> about the low throughput at the very moment when resilvering is in progress.
>
> If the "perspective" of "zpool status" is correct, then I have to wait after two hours
> for another 100 hours - ~ 4 days? Ups ... I think there is something badly misconfigured
> or missing.
>
> The pool is not "tuned" in any very sophisticated way, since I trust the kernels ability
> to scale automatically - if I'm not wrong for amd64 ...
>
> Are there any aspects to look at?
>
> Thanks in advance,
>
> Oliver
>
> --
> O. Hartmann
>
> Ich widerspreche der Nutzung oder Übermittlung meiner Daten für
> Werbezwecke oder für die Markt- oder Meinungsforschung (§ 28 Abs. 4 BDSG).


This is kind of to be expected - for whatever reason, resilvers seem
to go super slow at first and then speed up significantly. Just don't
ask me how long "at first" is - I'd give it several (more) hours.
I don't _think_ sync affects resilver speed (since that's a filesystem
level setting and the resilver happens at the block level), but that's
pure speculation.
The Tuning guide at https://wiki.freebsd.org/ZFSTuningGuide also
suggests a list of sysctls you can tweak, which could be worth a try:

### wiki
If you're getting horrible performance during a scrub or resilver, the
following sysctls can be set:

vfs.zfs.scrub_delay=0
vfs.zfs.top_maxinflight=128
vfs.zfs.resilver_min_time_ms=5000
vfs.zfs.resilver_delay=0

Setting those sysctls to those values increased my (Shawn Webb's)
resilver performance from 7MB/s to 230MB/s.
###



-- 
Daniel Nebdal