Re: ZFS L2ARC - incorrect size and abnormal system load on r255173

Hm, another strange thing's on another server -

zfs-stats -L

------------------------------------------------------------------------
ZFS Subsystem Report                            Thu Oct 10 12:56:54 2013
------------------------------------------------------------------------

L2 ARC Summary: (DEGRADED)
        Passed Headroom:                        8.34m
        Tried Lock Failures:                    145.66m
        IO In Progress:                         9.76m
        Low Memory Aborts:                      526
        Free on Write:                          1.70m
        Writes While Full:                      29.28k
        R/W Clashes:                            341.30k
        Bad Checksums:                          865.91k
        IO Errors:                              44.19k
        SPA Mismatch:                           32.03m

L2 ARC Size: (Adaptive)                         189.28  GiB
        Header Size:                    4.88%   9.24    GiB

Looks like size have nothing similar with IO errors.

So question is - when error's like Bad Checksums and IO Errors can hapend?

Look's like no hardware problem's.

All ssd atached to onboard intel sata controler (Motherboard  is Supermicro X9SRL-F) 
 

Vitalij Satanivskij wrote:
VS> Same situation hapend yesterday again :( 
VS> 
VS> What's confuse me while trying to understend where I'm wrong
VS> 
VS> 
VS> Firt some info.
VS> 
VS> We have zfs pool  "POOL" and one more zfs on it "POOL/zfs"
VS> 
VS> POOL - have only primarycache enabled "ALL" 
VS> POOL/zfs - have both primay and secondary for "ALL"
VS> 
VS> POOL have compression=lz4
VS> 
VS> POOL/zfs have none
VS> 
VS> 
VS> POOL - have around 9TB data
VS> 
VS> POOL/zfs - have 1TB
VS> 
VS> Secondary cache have configuration - 
VS> 
VS>         cache
VS>           gpt/cache0    ONLINE       0     0     0
VS>           gpt/cache1    ONLINE       0     0     0
VS>           gpt/cache2    ONLINE       0     0     0
VS> 
VS> gpt/cache0-2 it's intel sdd SSDSC2BW180A4 180gb 
VS> 
VS> So full real size  for l2 is 540GB (realy 489gb)
VS> 
VS> First question  - data on l2arc will be compressed on not?
VS> 
VS> Second in stats we see 
VS> 
VS> L2 ARC Size: (Adaptive)                         2.08    TiB
VS> 
VS> eary it was 1.1 1.4 ...
VS> 
VS> So a) how cache can be biger than zfs it self
VS>    b) in case it's not compressed (answer for first question) how it an be biger than real ssd size?
VS> 
VS> 
VS> one more coment if l2 arc size grove above phisical sizes I se  next stats 
VS> 
VS> kstat.zfs.misc.arcstats.l2_cksum_bad: 50907344
VS> kstat.zfs.misc.arcstats.l2_io_error: 4547377
VS> 
VS> and growing.
VS> 
VS> 
VS> System is r255173 with patch from rr255173
VS> 
VS> 
VS> At last maybe somebody have any ideas what's realy hapend...
VS> 
VS> 
VS> 
VS> 
VS> 
VS> Vitalij Satanivskij wrote:
VS> VS> 
VS> VS> One more question - 
VS> VS> 
VS> VS> we have two counter - 
VS> VS> 
VS> VS> kstat.zfs.misc.arcstats.l2_size: 1256609410560
VS> VS> kstat.zfs.misc.arcstats.l2_asize: 1149007667712
VS> VS> 
VS> VS> can anybody explain how to understand them i.e.  l2_asize - real used space on l2arc an l2_size - uncompressed size,
VS> VS> 
VS> VS> or maybe something else ?
VS> VS> 
VS> VS> 
VS> VS> 
VS> VS> Vitalij Satanivskij wrote:
VS> VS> VS> 
VS> VS> VS> Data on pool have compressratio around 1.4 
VS> VS> VS> 
VS> VS> VS> On diferent servers with same data type and load L2 ARC Size: (Adaptive) can be diferent 
VS> VS> VS> 
VS> VS> VS> for example 1.04    TiB vs  1.45    TiB
VS> VS> VS> 
VS> VS> VS> But it's all have same porblem  - grow in time.
VS> VS> VS> 
VS> VS> VS> 
VS> VS> VS> More stange for us - 
VS> VS> VS> 
VS> VS> VS> ARC: 80G Total, 4412M MFU, 5040M MRU, 76M Anon, 78G Header, 2195M Other
VS> VS> VS> 
VS> VS> VS> 78G header size and ubnormal - 
VS> VS> VS> 
VS> VS> VS> kstat.zfs.misc.arcstats.l2_cksum_bad: 210920592
VS> VS> VS> kstat.zfs.misc.arcstats.l2_io_error: 7362414
VS> VS> VS> 
VS> VS> VS> sysctl's growing avery second.
VS> VS> VS> 
VS> VS> VS> All part's of server (as hardware part's) in in normal state.
VS> VS> VS> 
VS> VS> VS> After reboot no problem's for some period untile cache size grow to some limit.
VS> VS> VS> 
VS> VS> VS> 
VS> VS> VS> 
VS> VS> VS> Mark Felder wrote:
VS> VS> VS> MF> On Mon, Oct 7, 2013, at 13:09, Dmitriy Makarov wrote:
VS> VS> VS> MF> > 
VS> VS> VS> MF> > How can L2 ARC Size: (Adaptive) be 1.44 TiB (up) with total physical size
VS> VS> VS> MF> > of L2ARC devices 490GB?
VS> VS> VS> MF> > 
VS> VS> VS> MF> 
VS> VS> VS> MF> http://svnweb.freebsd.org/base?view=revision&revision=251478
VS> VS> VS> MF> 
VS> VS> VS> MF> L2ARC compression perhaps?
VS> VS> VS> MF> _______________________________________________
VS> VS> VS> MF> freebsd-current_at_freebsd.org mailing list
VS> VS> VS> MF> http://lists.freebsd.org/mailman/listinfo/freebsd-current
VS> VS> VS> MF> To unsubscribe, send any mail to "freebsd-current-unsubscribe_at_freebsd.org"
VS> VS> VS> _______________________________________________
VS> VS> VS> freebsd-current_at_freebsd.org mailing list
VS> VS> VS> http://lists.freebsd.org/mailman/listinfo/freebsd-current
VS> VS> VS> To unsubscribe, send any mail to "freebsd-current-unsubscribe_at_freebsd.org"
VS> VS> _______________________________________________
VS> VS> freebsd-current_at_freebsd.org mailing list
VS> VS> http://lists.freebsd.org/mailman/listinfo/freebsd-current
VS> VS> To unsubscribe, send any mail to "freebsd-current-unsubscribe_at_freebsd.org"
VS> _______________________________________________
VS> freebsd-current_at_freebsd.org mailing list
VS> http://lists.freebsd.org/mailman/listinfo/freebsd-current
VS> To unsubscribe, send any mail to "freebsd-current-unsubscribe_at_freebsd.org"