ffs_blkfree: freeing free block -- ps, traces, fsck log

From: Robert Watson <rwatson_at_FreeBSD.org>
Date: Fri, 2 May 2003 18:16:16 -0400 (EDT)
I updated a pxe box to a recent -current, and applied it to a UFS
partition I had on disk.  I ran several parallel tars, an rm -Rf on one of
the tar extract targets, and a dd if=/dev/zero of=tmp on the partition,
and within a few minutes reproduced the nefarious ffs_blkfree() panic. 
Some debugging information as follows; I included stack traces of some of
the more interesting threads.  I've also included the fsck output below --
the background file system checker was not active at the time as it's
manually mounted and fscked when used; I believe the file system has never
actually had the background file system checker used on it, certainly not
with a recent kernel.  The output from fsck -y on the partition is also
attached.  As you can see, there are some alarming "ALLOCATED FRAG xxx
MARKED FREE" messages.

Robert N M Watson             FreeBSD Core Team, TrustedBSD Projects
robert_at_fledge.watson.org      Network Associates Laboratories

tar: src/sys/i386/compile/GENERIC/opt_vmpage.h: Cannot open: File exists
tar:
src/sys/i386/compile/GENERIC/modules/data/stock/src/sys/modules/msdosfs/msd
osfs_lookup.o: Cannot utime: No such file or directory
dev = ad0s1c, block = 713128, fs = /mnt
panic: ffs_blkfree: freeing free block
Debugger("panic")
Stopped at      Debugger+0x54:  xchgl   %ebx,in_Debugger.0
db> trace
Debugger(c0503178,c059e560,c05144ae,c8c75b48,1) at Debugger+0x54
panic(c05144ae,c19ab490,ae1a8,0,c19188d4) at panic+0xab
ffs_blkfree(c1918800,c1b6d7fc,ae1a8,0,4000) at ffs_blkfree+0x4da
handle_workitem_freeblocks(c205a900,0,2,c05a0b08,c1a27c00) at
handle_workitem_fr
eeblocks+0x31a
process_worklist_item(0,0,3eb2e464,0,c1a66c24) at
process_worklist_item+0x20a
softdep_process_worklist(0,0,c050a328,6db,0) at
softdep_process_worklist+0xc0
sched_sync(0,c8c75d48,c0500e21,310,0) at sched_sync+0x2ee
fork_exit(c036e9c0,0,c8c75d48) at fork_exit+0xc0
fork_trampoline() at fork_trampoline+0x1a
--- trap 0x1, eip = 0, esp = 0xc8c75d7c, ebp = 0 ---
db> ps
  pid   proc     addr    uid  ppid  pgrp  flag   stat  wmesg    wchan  cmd
  604 c1b4a780 c91d3000    0     0     0 0000204 norm[SLPQ  nfsidl
c05ca914][SLP] nfsiod 5
  603 c1b4a3c0 c91d1000    0     0     0 0000204 norm[SLPQ  nfsidl
c05ca910][SLP] nfsiod 4
  594 c1b4ab40 c91d5000    0   492   594 0004002 norm[SLPQ  wdrain
c05c4178][SLP] dd
  589 c1a405a0 c9154000    0   492   588 0004002 norm[SLPQ   biord
c3f31fc0][SLP] tar
  580 c1b4a1e0 c91d0000    0   492   579 0015002 norm[SLPQ   biord
c3f840e0][SLP] tar
  579 c1b4a000 c91cf000    0   492   579 0015002 norm[SLPQ  pipdwt
c1c68370][SLP] tar
  578 c1a46780 c91ac000    0   492   577 0015002 norm[SLPQ   biord
c3f39d38][SLP] tar
  492 c1b4ad20 c91d6000    0   486   492 0004002 norm[SLPQ   ttyin
c19a6610][SLP] csh
  491 c1b4b000 c91d7000  425   442   442 0000100 norm[SLPQ  accept
c19b3d36][SLP] httpd
  490 c1b4b1e0 c91ff000  425   442   442 0000100 norm[SLPQ  accept
c19b3d36][SLP] httpd
  489 c1b4b3c0 c9200000  425   442   442 0000100 norm[SLPQ  accept
c19b3d36][SLP] httpd
  488 c1b4b5a0 c9201000  425   442   442 0000100 norm[SLPQ  accept
c19b3d36][SLP] httpd
  487 c1a463c0 c91aa000  425   442   442 0000100 norm[SLPQ  accept
c19b3d36][SLP] httpd
  486 c1915780 c8ccf000    0     1   486 0004102 norm[SLPQ    wait
c1915780][SLP] login
  485 c19ad780 c8cfe000    0     1   485 0004002 norm[SLPQ   ttyin
c1a27610][SLP] getty
  484 c1a40000 c914a000    0     1   484 0004002 norm[SLPQ   ttyin
c1a27210][SLP] getty
  483 c1a403c0 c9153000    0     1   483 0004002 norm[SLPQ   ttyin
c1a26e10][SLP] getty
  482 c1a401e0 c9152000    0     1   482 0004002 norm[SLPQ   ttyin
c18d9c10][SLP] getty
  481 c1a461e0 c91a9000    0     1   481 0004002 norm[SLPQ   ttyin
c18d9410][SLP] getty
  480 c1a40d20 c91a7000    0     1   480 0004002 norm[SLPQ   ttyin
c1a26810][SLP] getty
  479 c1a40960 c9157000    0     1   479 0004002 norm[SLPQ   ttyin
c1a26410][SLP] getty
  478 c1a40b40 c9158000    0     1   478 0004002 norm[SLPQ   ttyin
c1906010][SLP] getty
  459 c19add20 c8d01000    0     1   459 0000000 norm[CVQ  select
c05c3d74][SLP] inetd
  442 c1a46000 c91a8000    0     1   442 0000000 norm[CVQ  select
c05c3d74][SLP] httpd
  420 c1a465a0 c91ab000   25     1   420 0000100 norm[SLPQ   pause
c91ab000][SLP] sendmail
  417 c1915960 c8cd0000    0     1   417 0000100 norm[CVQ  select
c05c3d74][SLP] sendmail
  411 c19ad1e0 c8cfb000    0     1   411 0000100 norm[CVQ  select
c05c3d74][SLP] sshd
  401 c19ad5a0 c8cfd000    0     1   401 0000000 norm[CVQ  select
c05c3d74][SLP] usbd
  344 c1915b40 c8cf8000    0     1   344 0000000 norm[CVQ  select
c05c3d74][SLP] rpc.statd
  290 c19153c0 c8ccd000    0     1   290 0000000 norm[CVQ  select
c05c3d74][SLP] ypbind
  275 c19155a0 c8cce000    0     1   275 0000000 norm[CVQ  select
c05c3d74][SLP] rpcbind
  209 c19ad000 c8cfa000    0     1   209 0000000 norm[CVQ  select
c05c3d74][SLP] dhclient
  124 c19ad3c0 c8cfc000    0     1   124 0000000 norm[SLPQ   pause
c8cfc000][SLP] adjkerntz
   44 c19ad960 c8cff000    0     0     0 0000204 norm[SLPQ  nfsidl
c05ca90c][SLP] nfsiod 3
   43 c19adb40 c8d00000    0     0     0 0000204 norm[SLPQ  nfsidl
c05ca908][SLP] nfsiod 2
   42 c189bb40 c8c97000    0     0     0 0000204 norm[SLPQ  nfsidl
c05ca904][SLP] nfsiod 1
   41 c189bd20 c8c98000    0     0     0 0000204 norm[SLPQ  nfsidl
c05ca900][SLP] nfsiod 0
   40 c1913000 c8cc3000    0     0     0 0000204 norm[SLPQ  vlruwt
c1913000][SLP] vnlru
   39 c19131e0 c8cc4000    0     0     0 0000204 norm[CPU 0] syncer
   38 c19133c0 c8cc5000    0     0     0 0000204 norm[SLPQ  wdrain
c05c4178][SLP] bufdaemon
   37 c19135a0 c8cc6000    0     0     0 000020c norm[SLPQ  pgzero
c05cbfc8][SLP] pagezero
    9 c1913780 c8cc7000    0     0     0 0000204 norm[SLPQ  psleep
c05cbff4][SLP] vmdaemon
    8 c1913960 c8cc8000    0     0     0 0000204 norm[SLPQ  psleep
c05cbfe0][SLP] pagedaemon
   36 c1913b40 c8cc9000    0     0     0 0000204 new [IWAIT] irq8: rtc
   35 c1913d20 c8cca000    0     0     0 0000204 new [IWAIT] irq0: clk
   34 c1915000 c8ccb000    0     0     0 0000204 new [IWAIT] irq3: sio1
   33 c19151e0 c8ccc000    0     0     0 0000204 new [IWAIT] irq4: sio0
   32 c17c85a0 c8c5f000    0     0     0 0000204 norm[IWAIT] swi0: tty:sio
   31 c17c8780 c8c60000    0     0     0 0000204 norm[IWAIT] irq6: fdc0
   30 c17c8960 c8c61000    0     0     0 0000204 new [IWAIT] irq5: fxp3
   29 c17c8b40 c8c62000    0     0     0 0000204 new [IWAIT] irq7: fxp2
   28 c17c8d20 c8c63000    0     0     0 0000204 norm[SLPQ  usbtsk
c05958b8][SLP] usbtask
   27 c189b000 c8c6a000    0     0     0 0000204 norm[SLPQ  usbevt
c18f0210][SLP] usb0
   26 c189b1e0 c8c6b000    0     0     0 0000204 new [IWAIT] irq11: ohci0
   25 c189b3c0 c8c6c000    0     0     0 0000204 norm[IWAIT] irq15: ata1
   24 c189b5a0 c8c6d000    0     0     0 0000204 norm[RUNQ] irq14: ata0
   23 c189b780 c8c6e000    0     0     0 0000204 norm[IWAIT] irq10: fxp0
fxp1
    7 c189b960 c8c6f000    0     0     0 0000204 norm[SLPQ  actask
c06e44ec][SLP] acpi_task2
    6 c0b8b1e0 c866d000    0     0     0 0000204 norm[SLPQ  actask
c06e44ec][SLP] acpi_task1
    5 c0b8b3c0 c866e000    0     0     0 0000204 norm[SLPQ  actask
c06e44ec][SLP] acpi_task0
   22 c0b8b5a0 c866f000    0     0     0 0000204 new [IWAIT] irq9: acpi0
   21 c0b8b780 c8670000    0     0     0 0000204 new [IWAIT] irq13:
   20 c0b8b960 c8671000    0     0     0 0000204 new [IWAIT] swi5:+
   19 c0b8bb40 c8672000    0     0     0 0000204 new [IWAIT] swi6: task
queue
   18 c0b8bd20 c869a000    0     0     0 0000204 new [IWAIT] swi3: cambio
   17 c17c8000 c8c5c000    0     0     0 0000204 new [IWAIT] swi2: camnet
   16 c17c81e0 c8c5d000    0     0     0 0000204 new [IWAIT] swi6:
acpitaskq
   15 c17c83c0 c8c5e000    0     0     0 0000204 norm[SLPQ   sleep
c0589700][SLP] random
    4 c0b84000 c8616000    0     0     0 0000204 norm[SLPQ  g_down
c0597e54][SLP] g_down
    3 c0b841e0 c8665000    0     0     0 0000204 norm[SLPQ    g_up
c0597e50][SLP] g_up
    2 c0b843c0 c8666000    0     0     0 0000204 norm[SLPQ  g_events
c0597e48][SLP] g_event
   14 c0b845a0 c8667000    0     0     0 0000204 new [IWAIT] swi4: vm
   13 c0b84780 c8668000    0     0     0 000020c norm[RUNQ] swi7: tty:sio
clock
   12 c0b84960 c8669000    0     0     0 0000204 norm[IWAIT] swi1: net
   11 c0b84b40 c866a000    0     0     0 000020c norm[Can run] idle
    1 c0b84d20 c866b000    0     0     1 0004200 norm[SLPQ    wait
c0b84d20][SLP] init
   10 c0b8b000 c866c000    0     0     0 0000204 norm[CVQ  ktrace
c059b3f4][SLP] ktrace
    0 c0597f20 c0713000    0     0     0 0000200 norm[SLPQ   sched
c0597f20][SLP] swapper
db> trace 39
Debugger(c0503178,c059e560,c05144ae,c8c75b48,1) at Debugger+0x54
panic(c05144ae,c19ab490,ae1a8,0,c19188d4) at panic+0xab
ffs_blkfree(c1918800,c1b6d7fc,ae1a8,0,4000) at ffs_blkfree+0x4da
handle_workitem_freeblocks(c205a900,0,2,c05a0b08,c1a27c00) at
handle_workitem_fr
eeblocks+0x31a
process_worklist_item(0,0,3eb2e464,0,c1a66c24) at
process_worklist_item+0x20a
softdep_process_worklist(0,0,c050a328,6db,0) at
softdep_process_worklist+0xc0
sched_sync(0,c8c75d48,c0500e21,310,0) at sched_sync+0x2ee
fork_exit(c036e9c0,0,c8c75d48) at fork_exit+0xc0
fork_trampoline() at fork_trampoline+0x1a
        --- trap 0x1, eip = 0, esp = 0xc8c75d7c, ebp = 0
db> trace 38
mi_switch(c189c000,44,c05038db,ca,c8c72bc8) at mi_switch+0x219
msleep(c05c4178,c05c4180,44,c0508c44,0) at msleep+0x432
bwrite(c3f28f60,c8c72c68,c03658c1,c3f28f60,c3f29090) at bwrite+0x442
bawrite(c3f28f60,c3f29090,8,3c4,20020080) at bawrite+0x1c
cluster_wbuild(c1d69920,4000,0,0,2) at cluster_wbuild+0x6d1
vfs_bio_awrite(c3fb6a68,0,c0508b1f,87b,0) at vfs_bio_awrite+0x1f9
flushbufqueues(0,0,c0508b1f,10e,64) at flushbufqueues+0x179
buf_daemon(0,c8c72d48,c0500e21,310,0) at buf_daemon+0xdc
fork_exit(c035f600,0,c8c72d48) at fork_exit+0xc0
fork_trampoline() at fork_trampoline+0x1a
--- trap 0x1, eip = 0, esp = 0xc8c72d7c, ebp = 0 ---
db> trace 578
mi_switch(c1a41980,4c,c05038db,ca,c1b6d7fc) at mi_switch+0x219
msleep(c3f39d38,c05c4200,4c,c0509100,0) at msleep+0x432
bwait(c3f39d38,4c,c0509100,0,c91738b8) at bwait+0x70
bufwait(c3f39d38,0,0,800,0) at bufwait+0x38
breadn(c2009124,0,0,800,0) at breadn+0x2a2
bread(c2009124,0,0,800,0) at bread+0x4c
ffs_blkatoff(c2009124,0,0,0,c9173988) at ffs_blkatoff+0xcf
ufs_lookup(c9173ab8,c9173af4,c03636ec,c9173ab8,c9173c38) at
ufs_lookup+0x387
ufs_vnoperate(c9173ab8,c9173c38,c9173c4c,c0557fcc,c1a41980) at
ufs_vnoperate+0x1
8
vfs_cache_lookup(c9173b68,c9173b94,c03683a2,c9173b68,20002) at
vfs_cache_lookup+
0x2fc
ufs_vnoperate(c9173b68,20002,c1a41980,3,c1a41980) at ufs_vnoperate+0x18
lookup(c9173c24,0,c0509c4e,a4,c1a41980) at lookup+0x302
namei(c9173c24,0,c05a77a0,0,0) at namei+0x24e
lstat(c1a41980,c9173d10,c051d37a,3fb,2) at lstat+0x52
syscall(80e002f,2f,bfbf002f,8129c40,812c680) at syscall+0x26e
Xint0x80_syscall() at Xint0x80_syscall+0x1d
--- syscall (190, FreeBSD ELF32, lstat), eip = 0x806204b, esp =
0xbfbffb3c, ebp
= 0xbfbffbe8 ---
db> trace 580
mi_switch(c1a41390,4c,c05038db,ca,c1b6d7fc) at mi_switch+0x219
msleep(c3f840e0,c05c4200,4c,c0509100,0) at msleep+0x432
bwait(c3f840e0,4c,c0509100,800,c9164a14) at bwait+0x70
bufwait(c3f840e0,0,0,800,0) at bufwait+0x38
breadn(c1e48a44,0,0,800,0) at breadn+0x2a2
bread(c1e48a44,0,0,800,0) at bread+0x4c
ffs_read(c9164b10,c031e554,c05c3cb4,c9164b98,c) at ffs_read+0x3ef
vn_rdwr(0,c1e48a44,c9164b98,c,0) at vn_rdwr+0x136
ufs_dirempty(c1da53f0,585c8,c1b5ad00,c9164bec,c9164c00) at
ufs_dirempty+0x8a
ufs_rmdir(c9164c20,c9164ccc,c03776ae,c9164c20,c9164c1c) at ufs_rmdir+0x55
ufs_vnoperate(c9164c20,c9164c1c,2,c1a27c00,c0573ee0) at ufs_vnoperate+0x18
kern_rmdir(c1a41390,80db7a0,0,c9164d40,c049e1fe) at kern_rmdir+0x1ae
rmdir(c1a41390,c9164d10,c051d37a,3fb,1) at rmdir+0x22
syscall(80e002f,2f,bfbf002f,80db7b8,80db7a0) at syscall+0x26e
Xint0x80_syscall() at Xint0x80_syscall+0x1d
--- syscall (137, FreeBSD ELF32, rmdir), eip = 0x80620af, esp =
0xbfbff93c, ebp
= 0xbfbff948 ---
db> trace 589
mi_switch(c189c980,4c,c05038db,ca,b824000) at mi_switch+0x219
msleep(c3f31fc0,c05c4200,4c,c0509100,0) at msleep+0x432
bwait(c3f31fc0,4c,c0509100,0,c8c8a864) at bwait+0x70
bufwait(c3f31fc0,0,0,4000,0) at bufwait+0x38
breadn(c1b6d7fc,5c120,0,4000,0) at breadn+0x2a2
bread(c1b6d7fc,5c120,0,4000,0) at bread+0x4c
ffs_vget(c1a27c00,6077,2,c8c8a984,c8c8a988) at ffs_vget+0x2dd
ufs_lookup(c8c8aab8,c8c8aaf4,c03636ec,c8c8aab8,c8c8ac38) at
ufs_lookup+0xdfb
ufs_vnoperate(c8c8aab8,c8c8ac38,c8c8ac4c,c0557fcc,c189c980) at
ufs_vnoperate+0x1
8
vfs_cache_lookup(c8c8ab68,c8c8ab94,c03683a2,c8c8ab68,20002) at
vfs_cache_lookup+
0x2fc
ufs_vnoperate(c8c8ab68,20002,c189c980,3,c189c980) at ufs_vnoperate+0x18
lookup(c8c8ac24,0,c0509c4e,a4,c189c980) at lookup+0x302
namei(c8c8ac24,0,c05a77a0,0,0) at namei+0x24e
stat(c189c980,c8c8ad10,c051d37a,3fb,2) at stat+0x52
syscall(2f,80e002f,bfbf002f,8123a71,3) at syscall+0x26e
Xint0x80_syscall() at Xint0x80_syscall+0x1d
--- syscall (188, FreeBSD ELF32, stat), eip = 0x8062073, esp = 0xbfbffb3c,
ebp =
 0xbfbffbe8 ---
db> trace 594
mi_switch(c1916d10,44,c05038db,ca,c8cf49c8) at mi_switch+0x219
msleep(c05c4178,c05c4180,44,c0508c44,0) at msleep+0x432
bwrite(c3f292d0,c8cf4a68,c03658c1,c3f292d0,c3f29400) at bwrite+0x442
bawrite(c3f292d0,c3f29400,20,3c4,20020080) at bawrite+0x1c
cluster_wbuild(c1ca25b4,4000,acaa,0,8) at cluster_wbuild+0x6d1
cluster_write(c3f8d140,2b2c8000,0,7f,c1b5ad00) at cluster_write+0x591
ffs_write(c8cf4be0,20002,c1916d10,227,7f0001) at ffs_write+0x56c
vn_write(c18d1f3c,c8cf4c7c,c1b5ad00,0,c1916d10) at vn_write+0x222
dofilewrite(c1916d10,c18d1f3c,4,8062000,200) at dofilewrite+0xe8
write(c1916d10,c8cf4d10,c051d37a,3fb,3) at write+0x69
syscall(806002f,2f,bfbf002f,8062000,200) at syscall+0x26e
Xint0x80_syscall() at Xint0x80_syscall+0x1d
--- syscall (4, FreeBSD ELF32, write), eip = 0x804aec3, esp = 0xbfbffb9c,
ebp =
0xbfbffbc8 ---
db>    show lockedvnods
Locked vnodes
0xc1ca25b4: tag ufs, type VREG, usecount 2, writecount 1, refcount 530,
flags (VV_OBJBUF), lock type ufs: EXCL (count 1) by thread 0xc1916d10
        ino 333312, on dev ad0s1c (4, 14)
0xc1daeb68: tag ufs, type VDIR, usecount 1, writecount 0, refcount 2, lock
type ufs: EXCL (count 1) by thread 0xc189c980
        ino 518501, on dev ad0s1c (4, 14)
0xc1d69920: tag ufs, type VREG, usecount 0, writecount 0, refcount 3,
flags (VV_OBJBUF), lock type ufs: EXCL (count 1) by thread 0xc189c000
        ino 389331, on dev ad0s1c (4, 14)
0xc20a036c: tag ufs, type VDIR, usecount 1, writecount 0, refcount 1, lock
type ufs: EXCL (count 1) by thread 0xc1a41390
        ino 361928, on dev ad0s1c (4, 14)
0xc2009124: tag ufs, type VDIR, usecount 1, writecount 0, refcount 1, lock
type ufs: EXCL (count 1) by thread 0xc1a41980
        ino 101605, on dev ad0s1c (4, 14)
0xc1e48a44: tag ufs, type VDIR, usecount 1, writecount 0, refcount 1, lock
type ufs: EXCL (count 1) by thread 0xc1a41390
        ino 361929, on dev ad0s1c (4, 14)
0xc1b75b68: tag ufs, type VNON, usecount 1, writecount 0, refcount 0, lock
type ufs: EXCL (count 1) by thread 0xc189c980
        ino 24695, on dev ad0s1c (4, 14)


** /dev/ad0s1a
** Last Mounted on /mnt
** Phase 1 - Check Blocks and Sizes
INCORRECT BLOCK COUNT I=294273 (4 should be 0)
CORRECT? yes

INCORRECT BLOCK COUNT I=294274 (4 should be 0)
CORRECT? yes

INCORRECT BLOCK COUNT I=294281 (4 should be 0)
CORRECT? yes

INCORRECT BLOCK COUNT I=294282 (4 should be 0)
CORRECT? yes

INCORRECT BLOCK COUNT I=294289 (4 should be 0)
CORRECT? yes

INCORRECT BLOCK COUNT I=294290 (4 should be 0)
CORRECT? yes

INCORRECT BLOCK COUNT I=294297 (4 should be 0)
CORRECT? yes

PARTIALLY TRUNCATED INODE I=333312
SALVAGE? yes

** Phase 2 - Check Pathnames
** Phase 3 - Check Connectivity
UNREF DIR  I=294298  OWNER=rwatson MODE=40755
SIZE=512 MTIME=Dec  2 19:39 2002 
RECONNECT? yes

NO lost+found DIRECTORY
CREATE? yes

DIR I=294298 CONNECTED. PARENT WAS I=294297

** Phase 4 - Check Reference Counts
LINK COUNT DIR I=291272  OWNER=rwatson MODE=40755
SIZE=3072 MTIME=May  1 22:22 2003  COUNT 173 SHOULD BE 164
ADJUST? yes

ZERO LENGTH DIR I=294273  OWNER=rwatson MODE=40755
SIZE=0 MTIME=Apr 16 23:41 2003 
CLEAR? yes

ZERO LENGTH DIR I=294274  OWNER=rwatson MODE=40755
SIZE=0 MTIME=Apr 16 23:41 2003 
CLEAR? yes

UNREF FILE  I=294275  OWNER=rwatson MODE=100644
SIZE=11 MTIME=Mar 27 13:48 2003 
RECONNECT? yes

UNREF FILE  I=294276  OWNER=rwatson MODE=100644
SIZE=23 MTIME=Dec  2 19:39 2002 
RECONNECT? yes

UNREF FILE  I=294277  OWNER=rwatson MODE=100644
SIZE=134 MTIME=Apr 16 23:41 2003 
RECONNECT? yes

UNREF FILE  I=294278  OWNER=rwatson MODE=100644
SIZE=172 MTIME=Apr 16 23:41 2003 
RECONNECT? yes

UNREF FILE  I=294279  OWNER=rwatson MODE=100644
SIZE=15621 MTIME=Apr 16 23:41 2003 
RECONNECT? yes

UNREF FILE  I=294280  OWNER=rwatson MODE=100644
SIZE=26885 MTIME=Apr 16 23:41 2003 
RECONNECT? yes

ZERO LENGTH DIR I=294281  OWNER=rwatson MODE=40755
SIZE=0 MTIME=Dec  2 19:39 2002 
CLEAR? yes

ZERO LENGTH DIR I=294282  OWNER=rwatson MODE=40755
SIZE=0 MTIME=Dec  2 19:39 2002 
CLEAR? yes

UNREF FILE  I=294283  OWNER=rwatson MODE=100644
SIZE=11 MTIME=Mar 27 13:48 2003 
RECONNECT? yes

UNREF FILE  I=294284  OWNER=rwatson MODE=100644
SIZE=22 MTIME=Dec  2 19:39 2002 
RECONNECT? yes

UNREF FILE  I=294285  OWNER=rwatson MODE=100644
SIZE=131 MTIME=Dec  2 19:39 2002 
RECONNECT? yes

UNREF FILE  I=294286  OWNER=rwatson MODE=100644
SIZE=138 MTIME=Jul 20 02:20 2001 
RECONNECT? yes

UNREF FILE  I=294287  OWNER=rwatson MODE=100644
SIZE=4104 MTIME=Jul 14 10:46 2002 
RECONNECT? yes

UNREF FILE  I=294288  OWNER=rwatson MODE=100644
SIZE=11889 MTIME=Nov 26 18:51 2000 
RECONNECT? yes

ZERO LENGTH DIR I=294289  OWNER=rwatson MODE=40755
SIZE=0 MTIME=Apr 16 23:41 2003 
CLEAR? yes

ZERO LENGTH DIR I=294290  OWNER=rwatson MODE=40755
SIZE=0 MTIME=Apr 16 23:41 2003 
CLEAR? yes

UNREF FILE  I=294291  OWNER=rwatson MODE=100644
SIZE=11 MTIME=Mar 27 13:48 2003 
RECONNECT? yes

UNREF FILE  I=294292  OWNER=rwatson MODE=100644
SIZE=21 MTIME=Dec  2 19:39 2002 
RECONNECT? yes

UNREF FILE  I=294293  OWNER=rwatson MODE=100644
SIZE=129 MTIME=Apr 16 23:41 2003 
RECONNECT? yes

UNREF FILE  I=294294  OWNER=rwatson MODE=100644
SIZE=342 MTIME=Apr 16 23:41 2003 
RECONNECT? yes

UNREF FILE  I=294295  OWNER=rwatson MODE=100644
SIZE=6052 MTIME=Jul 14 10:47 2002 
RECONNECT? yes

UNREF FILE  I=294296  OWNER=rwatson MODE=100644
SIZE=8051 MTIME=Apr 28 11:18 2002 
RECONNECT? yes

ZERO LENGTH DIR I=294297  OWNER=rwatson MODE=40755
SIZE=0 MTIME=Dec  2 19:39 2002 
CLEAR? yes

UNREF FILE  I=476464  OWNER=root MODE=100000
SIZE=0 MTIME=May  2 17:21 2003 
RECONNECT? yes

UNREF FILE  I=476677  OWNER=root MODE=100000
SIZE=0 MTIME=May  2 17:21 2003 
RECONNECT? yes

UNREF FILE  I=476678  OWNER=root MODE=100000
SIZE=0 MTIME=May  2 17:21 2003 
RECONNECT? yes

UNREF FILE  I=476686  OWNER=root MODE=100000
SIZE=0 MTIME=May  2 17:21 2003 
RECONNECT? yes

UNREF FILE  I=476687  OWNER=root MODE=100000
SIZE=0 MTIME=May  2 17:21 2003 
RECONNECT? yes

UNREF FILE  I=476695  OWNER=root MODE=100000
SIZE=0 MTIME=May  2 17:21 2003 
RECONNECT? yes

UNREF FILE  I=476696  OWNER=root MODE=100000
SIZE=0 MTIME=May  2 17:21 2003 
RECONNECT? yes

UNREF FILE  I=476726  OWNER=root MODE=100000
SIZE=0 MTIME=May  2 17:21 2003 
RECONNECT? yes

UNREF FILE  I=476727  OWNER=root MODE=100000
SIZE=0 MTIME=May  2 17:21 2003 
RECONNECT? yes

UNREF FILE  I=476742  OWNER=root MODE=100000
SIZE=0 MTIME=May  2 17:21 2003 
RECONNECT? yes

UNREF FILE  I=476743  OWNER=root MODE=100000
SIZE=0 MTIME=May  2 17:21 2003 
RECONNECT? yes

UNREF FILE  I=476757  OWNER=root MODE=100000
SIZE=0 MTIME=May  2 17:21 2003 
RECONNECT? yes

UNREF FILE  I=476758  OWNER=root MODE=100000
SIZE=0 MTIME=May  2 17:21 2003 
RECONNECT? yes

** Phase 5 - Check Cyl groups
FREE BLK COUNT(S) WRONG IN SUPERBLK
SALVAGE? yes

SUMMARY INFORMATION BAD
SALVAGE? yes

BLK(S) MISSING IN BIT MAPS
SALVAGE? yes

ALLOCATED FRAGS 691445-691447 MARKED FREE
ALLOCATED FRAGS 707488-707495 MARKED FREE
ALLOCATED FRAGS 708280-708287 MARKED FREE
ALLOCATED FRAGS 708360-708367 MARKED FREE
ALLOCATED FRAGS 709940-709943 MARKED FREE
ALLOCATED FRAGS 711284-711287 MARKED FREE
ALLOCATED FRAGS 711660-711663 MARKED FREE
ALLOCATED FRAGS 711707-711711 MARKED FREE
ALLOCATED FRAGS 711770-711775 MARKED FREE
ALLOCATED FRAGS 717753-717759 MARKED FREE
ALLOCATED FRAGS 717850-717855 MARKED FREE
ALLOCATED FRAGS 718108-718111 MARKED FREE
ALLOCATED FRAGS 718170-718175 MARKED FREE
ALLOCATED FRAGS 718465-718471 MARKED FREE
ALLOCATED FRAGS 718755-718759 MARKED FREE
ALLOCATED FRAGS 718853-718855 MARKED FREE
ALLOCATED FRAGS 719061-719063 MARKED FREE
ALLOCATED FRAGS 719147-719151 MARKED FREE
ALLOCATED FRAGS 719171-719175 MARKED FREE
ALLOCATED FRAGS 719277-719279 MARKED FREE
ALLOCATED FRAGS 719402-719407 MARKED FREE
ALLOCATED FRAGS 719557-719559 MARKED FREE
ALLOCATED FRAGS 719714-719719 MARKED FREE
ALLOCATED FRAGS 719772-719775 MARKED FREE
ALLOCATED FRAGS 719877-719879 MARKED FREE
ALLOCATED FRAGS 719957-719959 MARKED FREE
ALLOCATED FRAG 719965 MARKED FREE
ALLOCATED FRAGS 719998-719999 MARKED FREE
ALLOCATED FRAGS 720006-720007 MARKED FREE
ALLOCATED FRAGS 720013-720015 MARKED FREE
ALLOCATED FRAGS 720021-720023 MARKED FREE
ALLOCATED FRAGS 720090-720095 MARKED FREE
ALLOCATED FRAGS 720324-720327 MARKED FREE
ALLOCATED FRAGS 720420-720423 MARKED FREE
ALLOCATED FRAG 720471 MARKED FREE
ALLOCATED FRAGS 720516-720519 MARKED FREE
ALLOCATED FRAG 720535 MARKED FREE
ALLOCATED FRAGS 720651-720655 MARKED FREE
ALLOCATED FRAGS 720813-720815 MARKED FREE
ALLOCATED FRAGS 720925-720927 MARKED FREE
ALLOCATED FRAGS 720980-720983 MARKED FREE
ALLOCATED FRAGS 721085-721087 MARKED FREE
ALLOCATED FRAGS 721163-721167 MARKED FREE
ALLOCATED FRAGS 721205-721207 MARKED FREE
ALLOCATED FRAGS 721277-721279 MARKED FREE
ALLOCATED FRAGS 721490-721493 MARKED FREE
ALLOCATED FRAGS 721526-721527 MARKED FREE
ALLOCATED FRAGS 721795-721799 MARKED FREE
ALLOCATED FRAGS 721837-721839 MARKED FREE
ALLOCATED FRAGS 722085-722087 MARKED FREE
ALLOCATED FRAGS 722214-722215 MARKED FREE
ALLOCATED FRAGS 722387-722391 MARKED FREE
ALLOCATED FRAGS 722428-722431 MARKED FREE
ALLOCATED FRAGS 722579-722583 MARKED FREE
ALLOCATED FRAGS 722772-722775 MARKED FREE
ALLOCATED FRAGS 722854-722855 MARKED FREE
ALLOCATED FRAGS 723002-723007 MARKED FREE
ALLOCATED FRAGS 723110-723111 MARKED FREE
ALLOCATED FRAG 723223 MARKED FREE
ALLOCATED FRAGS 723250-723254 MARKED FREE
ALLOCATED FRAGS 723342-723343 MARKED FREE
ALLOCATED FRAG 723471 MARKED FREE
ALLOCATED FRAGS 723523-723527 MARKED FREE
ALLOCATED FRAGS 723580-723583 MARKED FREE
ALLOCATED FRAGS 723733-723735 MARKED FREE
ALLOCATED FRAG 723927 MARKED FREE
ALLOCATED FRAGS 724094-724095 MARKED FREE
ALLOCATED FRAGS 724195-724199 MARKED FREE
ALLOCATED FRAG 724239 MARKED FREE
ALLOCATED FRAGS 724350-724351 MARKED FREE
ALLOCATED FRAG 724391 MARKED FREE
ALLOCATED FRAGS 724434-724439 MARKED FREE
ALLOCATED FRAGS 724523-724527 MARKED FREE
ALLOCATED FRAGS 724771-724775 MARKED FREE
ALLOCATED FRAGS 724869-724871 MARKED FREE
ALLOCATED FRAGS 724933-724935 MARKED FREE
ALLOCATED FRAGS 725019-725023 MARKED FREE
ALLOCATED FRAGS 725100-725103 MARKED FREE
ALLOCATED FRAGS 725365-725367 MARKED FREE
ALLOCATED FRAG 725671 MARKED FREE
ALLOCATED FRAGS 725843-725847 MARKED FREE
ALLOCATED FRAGS 726192-726199 MARKED FREE
ALLOCATED FRAGS 726403-726407 MARKED FREE
ALLOCATED FRAGS 726526-726527 MARKED FREE
ALLOCATED FRAGS 726725-726727 MARKED FREE
ALLOCATED FRAGS 726835-726839 MARKED FREE
ALLOCATED FRAGS 726910-726911 MARKED FREE
ALLOCATED FRAG 727239 MARKED FREE
ALLOCATED FRAGS 727366-727372 MARKED FREE
ALLOCATED FRAGS 727405-727407 MARKED FREE
ALLOCATED FRAG 727447 MARKED FREE
163356 files, 1682621 used, 346410 free (20698 frags, 40714 blocks, 1.0% fragmentation)

***** FILE SYSTEM MARKED CLEAN *****

***** FILE SYSTEM WAS MODIFIED *****
Received on Fri May 02 2003 - 13:16:14 UTC

This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:37:06 UTC