Re: Softupdates not preventing lengthy fsck

On Tuesday 12 April 2005 05:01 am, Don Lewis wrote:
> On 11 Apr, Kris Kennaway wrote:
> > On Mon, Apr 11, 2005 at 06:43:17PM -0700, Don Lewis wrote:
> >> On 11 Apr, Kris Kennaway wrote:
> >> > I'm seeing the following problem: on 6.0 machines which have had a lot
> >> > of FS activity in the past but are currently quiet, an unclean reboot
> >> > will require an hour or more of fscking and will end up clearing
> >> > thousands of inodes:
> >> >
> >> > [...]
> >> > /dev/da0s1e: UNREF FILE I=269731  OWNER=root MODE=100644
> >> > /dev/da0s1e: SIZE=8555 MTIME=Apr 18 02:29 2002  (CLEARED)
> >> >
> >> > /dev/da0s1e: UNREF FILE I=269741  OWNER=root MODE=100644
> >> > [...]
> >> >
> >> > It's as if dirty buffers aren't being written out properly, or
> >> > something.  Has anyone else seen this?
> >>
> >> This looks a lot like it could be a vnode refcnt leak.  Files won't get
> >> removed from the disk while they are still in use (the old unlink while
> >> open trick).  Could nullfs be a factor?
> >
> > Yes, I make extensive use of read-only nullfs.
> >
> > Kris (fsck still running)
>
> It would also be interesting to find out why fsck is taking so long to
> run.  I don't see anything obvious in the code.

One HUGE time factor in a fsck run is serial consoles.  Printing tens or 
hundreds of thousands of inode corrections at 9600 baud takes forever.  At 
work, we found that some fsck runs that would take 20+ hours could be reduced 
to 15-20 minutes by simply redirecting fsck output to /dev/null instead of 
the serial console.

At work, we experimented with a memory based logging process that buffered up 
its stdin and waited until the fs was writeable.

eg:
fsck -p 2>&1 | memlogger /var/log/fsck.log

Memlogger would malloc memory to hold fsck's output and periodically poll 
for /var/log to become writable.  (There was more to it than that, and I'm 
not sure that we figured out all the quirks to make it usable in the /etc/rc 
environment)

-Peter