Re: SU+J systems do not fsck themselves

On Wed, Dec 28, 2011 at 8:54 AM, Maxim Khitrov <max_at_mxcrypt.com> wrote:
> On Wed, Dec 28, 2011 at 11:42 AM, Matthias Andree
> <matthias.andree_at_gmx.de> wrote:
>> Am 27.12.2011 22:53, schrieb David Thiel:
>>> I've had multiple machines now (9.0-RC3, amd64, i386 and earlier
>>> 9-CURRENT on ppc) running SU+J that have had unexplained panics and
>>> crashes start happening relating to disk I/O. When I end up running a
>>> full fsck, it keeps turning out that the disk is dirty and corrupted,
>>> but no mechanism is in place with SU+J to detect and fix this. A bgfsck
>>> never happens, but a manual fsck in single-user does indeed fix the
>>> crashing and weird behavior. Others have tested their SU+J volumes and
>>> found them to have errors as well. This makes me super nervous.
>>
>> The one thing I figured is that in the light of power outages, or
>> crashing virtualization hosts, you really really really need to disable
>> disk write caches, and this affects softupdates, journalling, asynch
>> file systems, just about everything.
>>
>> The fact that makes matters worse is that journalling or softupdates
>> allow you to mount a silently-corrupted file system, whereas the
>> traditional UFS/UFS2 sync/asynch mounts will fsck themselves in the
>> foreground, so they get fixed before the FS panics.
>>
>> So can you be sure that:
>>
>> - your driver, chip set and hard disk execute ordered writes in order,

If they don't, it's a bug.  Not that there isn't buggy firmware out
there, but each layer of software does need to rely on the one below
actually doing what it's promised.

>> - your driver, chip set and hard disk actually write data to permanent
>> storage BEFORE acknowledging a successful write?

Not required by SU as they use an explicit BIO_FLUSH which should be
handled by the driver.

>> Whenever I fixed these issues, I had no more corruptions.
>>
>> For ata and sata, there are loader tunables you will want to set,
>> hw.ata.wc=0 and kern.cam.ada.write_cache=0.

This should not be necessary if the driver and firmware are not buggy.

>> If your drives are under ada, ad, or ahci related control, try these
>> settings.  For SCSI, use camcontrol to turn the write cache off.
>> softupdates is supposed to rectify most of the performance penalties
>> incurred.
>>
>> Note also that you needed to set ahci_load=YES and atapicam_load=YES in
>> 8.X, I've never bothered to check 7.X or 9.X WRT these settings.
>
> This is a bit off-topic, but I'm curious what the effect of NCQ is on
> softupdates? Since that too has the ability to reorder writes to disk,
> should it be disabled in addition to cache?

SU doesn't care about write ordering, as long as everything before a
BIO_FLUSH is really flushed by the time the BIO_FLUSH is acknowledged.

Cheers,
matthew

> Also, I would say that if you are using a hardware raid controller
> with a BBU, then allowing the use of controller's cache and write-back
> policy should be safe for use with softupdates. Any caching mechanism,
> for that matter, that has a separate power supply source should be ok.
> For example, the Intel 320 SSDs have a few on-board capacitors that
> are used to flush the cache in the event of a power loss.
>
> - Max
> _______________________________________________
> freebsd-current_at_freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-current
> To unsubscribe, send any mail to "freebsd-current-unsubscribe_at_freebsd.org"