Re: enabling inode hashes results in kernel panics

In message <20181213113925.5f100b5e_at_ernst.home>, Gary Jennejohn writes:
> On Thu, 13 Dec 2018 10:47:13 +0100
> Gary Jennejohn <gljennjohn_at_gmail.com> wrote:
>
> > I just had two panics as a result of enabling inode hashes on a
> > file system yesterday when I had to run fsck on it.
> > 
> > The filesystem showed no problems at all yesterday, the problem
> > didn't appear until I accessed it today.
> > 
> > NOTE that my fsck was installed from r341840 yesterday morning,
> > so it should be up-to-date.  My kernel is also at r341840.
> > 
> > I have run fsck on this filesystem six times.  Although fsck
> > claims to have repaired the inode-check hash every time, it in
> > reality has not repaired anything.  I know this because, after
> > the first two fsck runs, I mounted the filesystem.  Accessing it
> > resulted in a immediate panic (the second of the two).
> > 
> > The file system is located on a SSD with trim enabled.  Whether
> > that is relevant I cannot say, but this is the ONLY filesystem
> > with inode hashes enabled.
> > 
> > The inodes are all contiguous, from 122229888 thorugh 122229904. 
> > Strangely, MTIME=Sep 6 23:21 2002 on every one of them, but the
> > SSD itslef is only a month old and the files on it only a few
> > weeks old.
> > 
> > The panic message is
> > 	Inode 122229888: check-hash failed
> > 	panic: softdep_update_inodeblock: bad link count
> > 
> > It almost appears like the softdep code in the kernel is not
> > aware of inode hashes and gets confused.
> > 
> > I have the crash dumps and the core.txt files.
> > 
> > It doesn't appear that I can disable the hashes using tunefs, so
> > my only remedy will be to run fsdb and clear these inodes and lose
> > quite a few rather large files.
> > 
>
> I ran fsdb on every affected inode and allowed it to correct
> the inode hashes.
>
> After that, fsck no longer found any errors and I was able to
> mount and access the filesystem without a kernel panic.
>
> So, it would appear that fsdb really does correct inode hash
> errors, whereas fsck does not.

My testing in a VM indicates that only the rootfs is affected. Adding 
inode checksums to a non-root filesystem appears to work. Whereas 
adding inode checksums to / results in the panic. My test was performed 
on a VirtualBox VM, its disks being zvols. The fsck's were performed on 
the host system. Host and guest are running,

13.0-CURRENT FreeBSD 13.0-CURRENT #59 r342045M

During boot with inode check hashes enabled on all filesystems except 
rootfs the VM boots normally. With check hashes enabled on rootfs it 
panics. My next test was to mount the guest VM's rootfs on the host. It 
mounted without error.

However as you say, fsck never fixes the incorrect check hashes.

My next test was to add inode check hashes to the filesystem but never 
mount it, just run fsck against it on the host, not the guest. fsck 
didn't fix any check hashes.

I only had half an hour to look at this but my guess is that there are 
two separate issues here.


-- 
Cheers,
Cy Schubert <Cy.Schubert_at_cschubert.com>
FreeBSD UNIX:  <cy_at_FreeBSD.org>   Web:  http://www.FreeBSD.org

	The need of the many outweighs the greed of the few.