> A new transaction group (TXG) is created at LEAST every > vfs.zfs.txg.timeout (defaults to 5) seconds. > f you offline a drive for hours or more, it must have all blocks with a > 'birth time' newer than the last transaction that was recorded on the > offlined drive replayed to catch that drive up to the other drives in > the pool. > As long as you have enough redundancy, the checksum errors can be > corrected without concern. > In the end, the checksum errors can be written off as being caused by > the bad hardware. After you finish the scrub and everything is OK, do: > 'zpool clear poolname', and it will reset all of the error and checksum > counts to 0, so you can track if any more ever show up. Thanks Allan, can always count on you for crystal clear answers =]. I'm surprised tho that it would be concluded as bad hardware(assuming you mean hd?). Just seems like its too much of a coincidence. I always ran zpool clear each time after the resilver/scrub was completed. > Perhaps on or more of the drives running out of Realloc Sectors? > I had once a case where smartctl showed no issues but zfs scrubbing showed > a defect, some weeks later smartctl was showing some reallocated sectors > and one week later the HD was out of spare sectors. > Have you already tested every single HD for smart issues? Smartd is set to run a short test weekly on Tuesday Thursday and Saturday. Extended test is performed weekly on Tuesday an hour after the short test. This occurs on all 24 drives. A scrub is performed once per month on Saturday an hour after the short test. 5 Reallocated_Sector_Ct 0x0033 100 100 010 Pre-fail Always - 0 This is the value of Reallocated sectors on all the drives(I think this is the normal value?). This drives smart looks like the worst of the lot. === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED See vendor-specific Attribute list for marginal Attributes. General SMART Values: Offline data collection status: (0x82) Offline data collection activity was completed without error. Auto Offline Data Collection: Enabled. Self-test execution status: ( 0) The previous self-test routine completed without error or no self-test has ever been run. Total time to complete Offline data collection: ( 592) seconds. Offline data collection capabilities: (0x7b) SMART execute Offline immediate. Auto Offline data collection on/off support. Suspend Offline collection upon new command. Offline surface scan supported. Self-test supported. Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 1) minutes. Extended self-test routine recommended polling time: ( 491) minutes. Conveyance self-test routine recommended polling time: ( 2) minutes. SCT capabilities: (0x50bd) SCT Status supported. SCT Error Recovery Control supported. SCT Feature Control supported. SCT Data Table supported. SMART Attributes Data Structure revision number: 10 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x000f 072 063 044 Pre-fail Always - 20189561 3 Spin_Up_Time 0x0003 091 091 000 Pre-fail Always - 0 4 Start_Stop_Count 0x0032 100 100 020 Old_age Always - 188 5 Reallocated_Sector_Ct 0x0033 100 100 010 Pre-fail Always - 0 7 Seek_Error_Rate 0x000f 092 085 030 Pre-fail Always - 1802626788 9 Power_On_Hours 0x0032 081 081 000 Old_age Always - 17457 10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail Always - 0 12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 158 184 End-to-End_Error 0x0032 100 100 099 Old_age Always - 0 187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0 188 Command_Timeout 0x0032 100 099 000 Old_age Always - 65537 189 High_Fly_Writes 0x003a 100 100 000 Old_age Always - 0 190 Airflow_Temperature_Cel 0x0022 055 045 045 Old_age Always In_the_past 45 (Min/Max 34/51) 191 G-Sense_Error_Rate 0x0032 100 100 000 Old_age Always - 0 192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 157 193 Load_Cycle_Count 0x0032 100 100 000 Old_age Always - 867 194 Temperature_Celsius 0x0022 045 055 000 Old_age Always - 45 (0 22 0 0 0) 195 Hardware_ECC_Recovered 0x001a 053 011 000 Old_age Always - 20189561 197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0 198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 0 SMART Error Log Version: 1 No Errors Logged SMART Self-test log structure revision number 1 Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error # 1 Extended offline Completed without error 00% 17423 - # 2 Short offline Completed without error 00% 17412 - # 3 Short offline Completed without error 00% 17340 - # 4 Short offline Completed without error 00% 17293 - # 5 Extended offline Completed without error 00% 17261 - # 6 Short offline Completed without error 00% 17245 - # 7 Short offline Completed without error 00% 17173 - # 8 Short offline Completed without error 00% 17125 - # 9 Extended offline Completed without error 00% 17101 - #10 Short offline Completed without error 00% 17084 - #11 Short offline Completed without error 00% 17012 - #12 Short offline Completed without error 00% 16964 - #13 Extended offline Completed without error 00% 16927 - #14 Short offline Completed without error 00% 16916 - #15 Short offline Completed without error 00% 16916 - #16 Short offline Completed without error 00% 16844 - #17 Short offline Completed without error 00% 16805 - #18 Extended offline Completed without error 00% 16775 - #19 Short offline Completed without error 00% 16757 - #20 Short offline Completed without error 00% 16685 - #21 Short offline Completed without error 00% 16637 - SMART Selective self-test log data structure revision number 1 SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS 1 0 0 Not_testing 2 0 0 Not_testing 3 0 0 Not_testing 4 0 0 Not_testing 5 0 0 Not_testing Selective self-test flags (0x0): After scanning selected spans, do NOT read-scan remainder of disk. If Selective self-test is pending on power-up, resume after 0 minute delay. On Wed, Aug 10, 2016 at 2:56 PM, olli hauer <ohauer_at_gmx.de> wrote: > On 2016-08-04 07:22, Ultima wrote: > > Hello, > > > > I recently had some issue with a PSU and ran several scrubs on a pool > with > > around 35T. Random drives would drop and require a zpool online, this > found > > checksum errors. (as expected) However, after all the scrubs I ran, I > think > > I may have found a bug with zpool online resilvering process. > > > > 24 disks total, 4 vdevs raidz2 (6 drives each). > > > > Before this next part... I had a backup PSU, however it was also going > bad > > and waiting for RMA. The current one seemed to be dieing but ran fine > with > > less drives. So I decided I would run the server short 4 drives. > > > > Started by offline(or already removed from psu) 4 drives from different > > vdevs, then ran a scrub to verify everything. Many sum errors were > present > > on some of the drives, but this was expected due to faulty psu. Then > > offlined 4 different drives and onlined the other 4 and scrubbed once > > again. After resilver, again, many sum errors on these drives as > expected. > > > > After the scrub completed, I decided to offline 4 different drives, then > > online the ones that were out of pool for awhile. During the resilver, > > checksum errors were once again found. I was surprised due to the recent > > scrub, So I decided to run another scrub, and it found even more checksum > > errors on these recently onlined drives. I didn't think much about it, > > however after the replacement PSU arrived, I onlined all the drives out > of > > pool and again, resilver had checksum errors as well as another scrub > with > > more sum errors. > > > > Is this issue known? Is it common for a scrub to be required after > onlining > > a disk that was out of pool for some time? > > > > The drives are ST4000NM0033, and until recent have never had a single > > checksum error in they're lifetime.(at least with zfs) > > FreeBSD S1 12.0-CURRENT FreeBSD 12.0-CURRENT #19 r303224: Sat Jul 23 > > 10:41:12 EDT 2016 > > root_at_S1:/usr/src/head/obj/usr/src/head/src/sys/MYKERNEL-NODEBUG > > amd64 > > > > > > Sorry for the wall of text, but I hope this helps in tracking down this > > possible bug. > > > > Perhaps on or more of the drives running out of Realloc Sectors? > I had once a case where smartctl showed no issues but zfs scrubbing showed > a defect, some weeks later smartctl was showing some reallocated sectors > and one week later the HD was out of spare sectors. > > Have you already tested every single HD for smart issues? > > -- > olli >Received on Wed Aug 10 2016 - 23:52:05 UTC
This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:41:07 UTC