Re: It still here... panic: ufs_dirbad: bad dir

From: Matthew Dillon <dillon_at_apollo.backplane.com>
Date: Fri, 17 Feb 2006 23:42:03 -0800 (PST)
:
:On Fri, 2006-Feb-17 20:39:53 -0800, Matthew Dillon wrote:
:>    I'm running out of ideas.  Right now my best idea is that there is
:>    something broken in the code that writes out the modified 'rewound'
:>    blocks.  Perhaps an old version of a buffer, with old already-reused 
:>    block pointers, is being written out and then something happens to 
:>    prevent the latest version from being written out.  I don't know, I'm
:>    grasping at straws here.  If I could only reliably reproduce the bug
:>    I would write some code to record every I/O operation done on the
:>    raw device then track back to the write that created the corruption.
:
:Is it worth setting up a ring buffer that just stores the last few
:thousand I/O requests and waiting for someone to trip over the panic?
:This should work if the corruption is close (in temporal terms) to
:the panic.
:
:-- 
:Peter Jeremy
    
    Only if the problem can be reproduced reliably.  The actual corruption
    is likely occuring on the order of tens of thousands or even millions
    of I/O's prior to the actual panic.   Valid 'clean' cached data is also
    probably hiding the corrupted on-disk blocks for a long period of time.

    The corruption is either:

    (1) The contents of a directory block is corrupted.
    (2) An indirect block pointer in the inode is corrupted.
    or 
    (3) The block pointers in the indirect block itself becomes corrupted.

    I have NEVER seen any corruption of a direct block for a directory.
    Not once.  I also have never seen any bitmap corruption (and I added
    a lot of sanity checks in the bitmap code).  The corruption I have
    seen tends to be data starting at the beginning of a block pointed to
    by an indirect block.

    This implies that either the indirect block as stored in the inode
    is wrong, that the block pointers in the indirect block are wrong,
    or that the data in those blocks is wrong.  If the leaf data were
    corrupted I would have expected the leaf data in the inode's direct
    blocks to be corrupted too, but I've never seen that. 

    The block arrays in the indirect blocks that I have examined have always
    looked like real block arrays.  i.e. one or two entries and then all
    zero's (due to the limited size of the directory).

    Perhaps softupdates is replacing the block array in an indirect block
    with an 'old' version due to indirect block dependancies and then never
    restoring it properly, somehow losing track of the mess.  I just don't
    know.  The corrupted data in the directories always looked like complete
    junk... part of some other file and not directory entries at all.  

    The only scenario that I can think of is that softupdates is storing
    'old' block pointers in the indirect block then losing track of the
    dependanc(ies) and never writing out the latest 'correct' block pointers.
    The old block pointers get reallocted to another purpose, such as a
    file, the 'clean' buffer containing the correct block pointers gets
    thrown away at some later point in time, and then the next access
    of that directory goes to disk, retrieves the 'bad' indirect block and
    related data, and panics.

					-Matt
					Matthew Dillon 
					<dillon_at_backplane.com>
Received on Sat Feb 18 2006 - 06:49:10 UTC

This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:38:52 UTC