Re: panic: ffs_blkfree: freeing free block

On Thursday 18 October 2007, Kostik Belousov wrote:
> On Wed, Oct 17, 2007 at 07:05:59AM -0500, Eric Anderson wrote:
> > Kostik Belousov wrote:
> > >On Tue, Oct 16, 2007 at 12:20:59PM +0200, Arjan van Leeuwen wrote:
> > >>2007/10/16, Eric Anderson <anderson_at_freebsd.org>:
> > >>>Pieter de Goeje wrote:
> > >>>>On Sunday 14 October 2007, Eric Anderson wrote:
> > >>>>>Pieter de Goeje wrote:
> > >>>>>>On Saturday 13 October 2007, Rolf Witt wrote:
> > >>>>>>>Backtrace from crash tonight (after portsnap cron):
> > >>>>>>
> > >>>>>><snip>
> > >>>>>>
> > >>>>>>># uname -a
> > >>>>>>>FreeBSD peanuts.homenet.local 7.0-PRERELEASE FreeBSD
> > >>>>>>> 7.0-PRERELEASE #121: Thu Oct 11 16:29:05 CEST 2007
> > >>>>>>>rowi_at_peanuts.homenet.local:/usr/obj/usr/src/sys/PEANUTS  i386
> > >>>>>>>
> > >>>>>>>Sources from Builddate.
> > >>>>>>>
> > >>>>>>>Rowi
> > >>>>>>
> > >>>>>>Eric sent me this patch:
> > >>>>>>http://www.googlebit.com/freebsd/patches/ffs_softdep.c-patch
> > >>>>>>which seems to be working great so far. I am still testing it, but
> > >>>>>> I think it fixed the problem.
> > >>>>>>To apply, cd /usr/src; patch < /path/to/patch and then rebuild the
> > >>>>>>kernel.
> > >>>>>>
> > >>>>>>Cheers,
> > >>>>>>Pieter de Goeje
> > >>>>>
> > >>>>>It doesn't actually 'fix' the problem, but I think it helps identify
> > >>>
> > >>>it.
> > >>>
> > >>>>>I'm not 100% certain if this is the right fix our not, but so far
> > >>>>>feedback has been good when running with this patch.
> > >>>>>
> > >>>>>Can somebody confirm that this patch is ok?
> > >>>>>
> > >>>>>Eric
> > >>>>
> > >>>>Can you elaborate on what this patch exactly does / what the problem
> > >>>> is?
> > >>>>
> > >>>>Pieter
> > >>>
> > >>>I think the problem is that blocks are being put in the worklist
> > >>> twice, but I'm not certain why yet.  The patch reduces the chance of
> > >>> this happening by more often removing the block from the worklist
> > >>> instead of leaving it on.  I think actually the patch is hiding the
> > >>> real issue more than anything, which is why I said it isn't fixing
> > >>> the problem, but because of the reports I've seen, I think we're
> > >>> sniffing in the right area.  I'll be looking more into this soon,
> > >>> when I get another few minutes of free time and my debugger..
> > >>
> > >>I actually seem to be getting the panic more often with the patch (i.e.
> > >>twice in just 30 minutes with the patch, about twice per day without
> > >> the patch), while doing the same things as usual. The panic usually
> > >> happens when
> > >>I'm compiling or linking. Is that strange?
> > >
> > >Patch is wrong. Actually, it does put the dirrem to the proccessing
> > > twice when xp != NULL.
> >
> > Yes, exactly why I said 'it doesn't fix the problem'.  In fact, I
> > actually did not post it to the list, and didn't intend for it to be
> > posted.  I was hoping for feedback for further examination.
> >
> > Kostik - do you have any ideas on what is going on here?  Did you see
> > the message in another thread that looked similar (it was regarding
> > gjournal)?
>
> No, I have no idea. BTW, some further information from the people that
> experience the problem could be helpful. For instance,
> UFS1/UFS2
UFS2

> size of the problematic fs/% of the space used
Filesystem  1K-blocks    Used     Avail Capacity  Mounted on
/dev/ad6s1f 143295066 8698604 123132858     7%    /usr

> are quotas enabled in kernel/active for the fs ?
No

> are softupdates on ?
Yes

> what block number is reported as being freed twice ? is it always the same
> ? what is the full fsck diagnostic after reboot ?
I'll try to get these items for you.
>
> The range of questions is so wide because I try to somewhat localize the
> search field down from "anywhere".
To me this problem happened on both SMP and UP systems, all running amd64.