Re: enabling inode hashes results in kernel panics

On Thu, 13 Dec 2018 07:45:36 -0800
Cy Schubert <Cy.Schubert_at_cschubert.com> wrote:

> In message <20181213113925.5f100b5e_at_ernst.home>, Gary Jennejohn writes:
> > On Thu, 13 Dec 2018 10:47:13 +0100
> > Gary Jennejohn <gljennjohn_at_gmail.com> wrote:
> >  
> > > I just had two panics as a result of enabling inode hashes on a
> > > file system yesterday when I had to run fsck on it.
> > > 
> > > The filesystem showed no problems at all yesterday, the problem
> > > didn't appear until I accessed it today.
> > > 
> > > NOTE that my fsck was installed from r341840 yesterday morning,
> > > so it should be up-to-date.  My kernel is also at r341840.
> > > 
> > > I have run fsck on this filesystem six times.  Although fsck
> > > claims to have repaired the inode-check hash every time, it in
> > > reality has not repaired anything.  I know this because, after
> > > the first two fsck runs, I mounted the filesystem.  Accessing it
> > > resulted in a immediate panic (the second of the two).
> > > 
> > > The file system is located on a SSD with trim enabled.  Whether
> > > that is relevant I cannot say, but this is the ONLY filesystem
> > > with inode hashes enabled.
> > > 
> > > The inodes are all contiguous, from 122229888 thorugh 122229904. 
> > > Strangely, MTIME=Sep 6 23:21 2002 on every one of them, but the
> > > SSD itslef is only a month old and the files on it only a few
> > > weeks old.
> > > 
> > > The panic message is
> > > 	Inode 122229888: check-hash failed
> > > 	panic: softdep_update_inodeblock: bad link count
> > > 
> > > It almost appears like the softdep code in the kernel is not
> > > aware of inode hashes and gets confused.
> > > 
> > > I have the crash dumps and the core.txt files.
> > > 
> > > It doesn't appear that I can disable the hashes using tunefs, so
> > > my only remedy will be to run fsdb and clear these inodes and lose
> > > quite a few rather large files.
> > >   
> >
> > I ran fsdb on every affected inode and allowed it to correct
> > the inode hashes.
> >
> > After that, fsck no longer found any errors and I was able to
> > mount and access the filesystem without a kernel panic.
> >
> > So, it would appear that fsdb really does correct inode hash
> > errors, whereas fsck does not.  
> 
> My testing in a VM indicates that only the rootfs is affected. Adding 
> inode checksums to a non-root filesystem appears to work. Whereas 
> adding inode checksums to / results in the panic. My test was performed 
> on a VirtualBox VM, its disks being zvols. The fsck's were performed on 
> the host system. Host and guest are running,
> 
> 13.0-CURRENT FreeBSD 13.0-CURRENT #59 r342045M
> 
> During boot with inode check hashes enabled on all filesystems except 
> rootfs the VM boots normally. With check hashes enabled on rootfs it 
> panics. My next test was to mount the guest VM's rootfs on the host. It 
> mounted without error.
> 
> However as you say, fsck never fixes the incorrect check hashes.
> 
> My next test was to add inode check hashes to the filesystem but never 
> mount it, just run fsck against it on the host, not the guest. fsck 
> didn't fix any check hashes.
> 
> I only had half an hour to look at this but my guess is that there are 
> two separate issues here.
> 

Thanks for looking into this.

In my case it was not the rootfs.  Mounting the filesystem for
the first time did not cause an error, it was the first access
which resulted in the panic.

You're running a more recent version than I am, but AFAIK no changes
were made to fsck post r341840.

-- 
Gary Jennejohn