Re: CURRENT: slow like crap! ZFS scrubbing and ports update > 25 min

Am Wed, 22 Mar 2017 21:10:51 +0100
Michael Gmelin <freebsd_at_grem.de> schrieb:

> > On 22 Mar 2017, at 21:02, O. Hartmann <ohartmann_at_walstatt.org> wrote:
> > 
> > CURRENT (FreeBSD 12.0-CURRENT #82 r315720: Wed Mar 22 18:49:28 CET 2017 amd64) is
> > annoyingly slow! While scrubbing is working on my 12 GB ZFS volume,
> > updating /usr/ports takes >25 min(!). That is an absolute record now.
> > 
> > I do an almost  daily update of world and ports tree and have periodic scrubbing ZFS
> > volumes every 35 days, as it is defined in /etc/defaults. Prts tree hasn't grown much,
> > the content of the ZFS volume hasn't changed much (~ 100 GB, its fill is about 4 TB
> > now) and this is now for ~ 2 years constant. 
> > 
> > I've experienced before that while scrubbing the ZFS volume, some operations, even the
> > update of /usr/ports which resides on that ZFS RAIDZ volume, takes a bit longer than
> > usual - but never that long like now!
> > 
> > Another box is quite unusable while it is scrubbing and it has been usable times
> > before. The change is dramatic ...
> >   
> 
> What do "zpool list", "gstat" and "zpool status" show?
> 
> 
> 
zpool list:

NAME       SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
TANK00  10.9T  5.45T  5.42T         -     7%    50%  1.58x  ONLINE  -

Deduplication is off right now, I had one ZFS filesystem with dedup enabled

gstat: not shown here, but the drives comprise the volume (4x 3 TB) show 100% busy each,
but one drive is always a bit off (by 10% lower) and this drive is walking through all
four drives ada2, ada3, ada4 and ada5. Nothing unusual in that situation. But the
throughput is incredible low, for example ada4:

 L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w   %busy Name
 2    174    174   1307   11.4      0      0    0.0   99.4| ada4

kBps (kilo Bits per second I presume) are peaking at ~ 4800 - 5000. On another bos, this
is ~ 20x higher! Most time, kBps r and w stay at ~ 500 -600.

zpool status:

  pool: TANK00
 state: ONLINE
  scan: scrub in progress since Wed Mar 22 19:12:41 2017
        167G scanned out of 5.62T at 15.9M/s, 99h46m to go
        0 repaired, 2.91% done
config:

        NAME              STATE     READ WRITE CKSUM
        TANK00          ONLINE       0     0     0
          raidz1-0        ONLINE       0     0     0
            gpt/tank00  ONLINE       0     0     0
            gpt/tank01  ONLINE       0     0     0
            gpt/tank02  ONLINE       0     0     0
            gpt/tank03  ONLINE       0     0     0
        logs
          gpt/zil00       ONLINE       0     0     0
        cache
          gpt/l2arc00     ONLINE       0     0     0

errors: No known data errors

Here are the incredible low numbers more clear. Usually, the hardware is capable of doing
~ 280 - 300 MBytes/s. It is stuck for hours at 15 - 16 MBytes/s.