Filesystem wedge, SUJ-related?

From: Gavin Atkinson <gavin_at_FreeBSD.org>
Date: Sat, 17 Jul 2010 18:32:45 +0100 (BST)
Hi guys,

Semi-regularly (every two-three days) I'm seeing what appears to be some 
sort of filesystem wedge.  I usually see it initially with web browsers, 
but it's possible that's only because it's what produces most disk 
activity on this machine.  I've seen it with both Opera and Firefox.

What happens is that the process will just wedge.  A "procstat -kk" on it 
shows the following stack backtrace:

 9012 100243 firefox-bin      initial thread   mi_switch+0x21d 
sleepq_switch+0x123 sleepq_wait+0x4d _sleep+0x357 getdirtybuf+0x21e 
flush_deplist+0x6f softdep_sync_metadata+0x153 ffs_syncvnode+0x213 
ffs_fsync+0x43 fsync+0x148 syscallenter+0x1b5 syscall+0x4c 
Xfast_syscall+0xe2 

 8954 100231 opera            initial thread   mi_switch+0x21d 
sleepq_switch+0x123 sleepq_wait+0x4d _sleep+0x357 getdirtybuf+0x21e 
flush_deplist+0x6f softdep_sync_metadata+0x153 ffs_syncvnode+0x213 
ffs_fsync+0x43 fsync+0x148 syscallenter+0x1b5 syscall+0x4c 
Xfast_syscall+0xe2

Reading from the disk is fine, indeed writing to the disk seems to work 
fine for other applications, even once a browser has wedged.

I've included the output of "sysctl -a | grep debug.softdep" later in the 
email, in case it gives any clues.  This was taken while the system was 
idle, about an hour after both Firefox and Opera had wedged.

As I say, this happens every few days, so please let me know what further 
information can be useful to diagnose this.  I can get info from the 
debugger too - but as I don't have much experience diagnosing FS issues 
somebody needs to tell me which info to obtain other than the standard 
(alltrace, ps, etc).

Thanks,

Gavin


debug.softdep.jwait_newblk: 3
debug.softdep.jwait_inode: 44
debug.softdep.jwait_freeblks: 115
debug.softdep.jwait_filepage: 181
debug.softdep.journal_wait: 15488
debug.softdep.journal_min: 0
debug.softdep.journal_low: 0
debug.softdep.jnewblk_rollback: 27953
debug.softdep.jaddref_rollback: 1811
debug.softdep.dir_entry: 23248
debug.softdep.direct_blk_ptrs: 5975
debug.softdep.inode_bitmap: 8631
debug.softdep.indir_blk_ptrs: 7670
debug.softdep.sync_limit_hit: 0
debug.softdep.ino_limit_hit: 0
debug.softdep.blk_limit_hit: 0
debug.softdep.ino_limit_push: 0
debug.softdep.blk_limit_push: 0
debug.softdep.worklist_push: 0
debug.softdep.maxindirdeps: 50
debug.softdep.tickdelay: 2
debug.softdep.max_softdeps: 400000
debug.softdep.current.jtrunc: 0
debug.softdep.current.sbdep: 0
debug.softdep.current.jsegdep: 5218
debug.softdep.current.jseg: 3921
debug.softdep.current.jfreefrag: 0
debug.softdep.current.jfreeblk: 0
debug.softdep.current.jnewblk: 0
debug.softdep.current.jmvref: 0
debug.softdep.current.jremref: 0
debug.softdep.current.jaddref: 11
debug.softdep.current.freedep: 18
debug.softdep.current.freework: 4666
debug.softdep.current.newdirblk: 0
debug.softdep.current.dirrem: 122
debug.softdep.current.mkdir: 0
debug.softdep.current.diradd: 132
debug.softdep.current.freefile: 2
debug.softdep.current.freeblks: 4198
debug.softdep.current.freefrag: 0
debug.softdep.current.allocindir: 0
debug.softdep.current.indirdep: 4
debug.softdep.current.allocdirect: 0
debug.softdep.current.newblk: 255
debug.softdep.current.bmsafemap: 3
debug.softdep.current.inodedep: 284
debug.softdep.current.pagedep: 119
debug.softdep.total.jtrunc: 114
debug.softdep.total.sbdep: 6109
debug.softdep.total.jsegdep: 489669
debug.softdep.total.jseg: 23468
debug.softdep.total.jfreefrag: 67993
debug.softdep.total.jfreeblk: 53638
debug.softdep.total.jnewblk: 180654
debug.softdep.total.jmvref: 160
debug.softdep.total.jremref: 95199
debug.softdep.total.jaddref: 92071
debug.softdep.total.freedep: 1044
debug.softdep.total.freework: 57287
debug.softdep.total.newdirblk: 267
debug.softdep.total.dirrem: 95173
debug.softdep.total.mkdir: 490
debug.softdep.total.diradd: 91573
debug.softdep.total.freefile: 67115
debug.softdep.total.freeblks: 38486
debug.softdep.total.freefrag: 67993
debug.softdep.total.allocindir: 0
debug.softdep.total.indirdep: 3798
debug.softdep.total.allocdirect: 0
debug.softdep.total.newblk: 180654
debug.softdep.total.bmsafemap: 11319
debug.softdep.total.inodedep: 197151
debug.softdep.total.pagedep: 89352
Received on Sat Jul 17 2010 - 16:04:36 UTC

This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:40:05 UTC