Re: How to help debugging of lock-up

From: Jun Kuriyama <kuriyama_at_imgsrc.co.jp>
Date: Fri, 17 Jun 2005 12:45:09 +0900
At Tue, 14 Jun 2005 19:48:27 -0700 (PDT),
Doug White wrote:
> > I'm not sure which process I should suspect.  Is there something I can
> > provide to help debugging about this?
> 
> The trace looks normal for something network- and disk-bound. Perhaps your
> NIC's overloaded or hung?  Where is the amanda backup going -- back to the
> same system?

Yes, it looks trace is normal.  But even serial console, getty does
not respond.

Backup server is another box, dump is going through the network.


And I got another lock-up with today's kernel (including jeff's r1.103
of vfs_cache.c).

If noone sees this behavior, is there a possibility which my hardware
is broken?  (but old 2005-04-09 kernel works without problem)


-----
 KDB: enter: Break sequence on console
[thread pid 11 tid 100006 ]
Stopped at      kdb_enter+0x2b: nop
db> ps
  pid   proc     uid  ppid  pgrp  flag   stat  wmesg    wchan  cmd
35659 c6569c00  103 35657   758 0004000 [SLPQ getblk 0xd7775df0][SLP] as
35657 c3939a00  103 35656   758 0004000 [SLPQ wait 0xc3939a00][SLP] c++
35656 c8d33c00  103 35635   758 0004000 [SLPQ wait 0xc8d33c00][SLP] sh
35655 c3b3cc00  103 35654   758 0004000 [SLPQ biord 0xd766c638][SLP] cc1
35654 c393c200  103 35653   758 0004000 [SLPQ wait 0xc393c200][SLP] cc
35653 c8ebe600  103 35651   758 0004000 [SLPQ wait 0xc8ebe600][SLP] sh
35651 c3b3ca00  103 35650   758 0004000 [SLPQ wait 0xc3b3ca00][SLP] sh
35650 c656a800  103 35350   758 0004000 [SLPQ select 0xc075cb24][SLP] make
35648 c8ec0e00    0 35646 35648 0000110 [SLPQ kqread 0xc8fd1480][SLP] cron
35647 c8d30000    0 35644 35647 0000010 [SLPQ kqread 0xc8ffc880][SLP] cron
35646 c953fa00    0   588   588 0000000 [SLPQ ppwait 0xc953fa00][SLP] cron
35644 c656a400    0   588   588 0000000 [SLPQ ppwait 0xc656a400][SLP] cron
35635 c8ebee00  103 34347   758 0004000 [SLPQ select 0xc075cb24][SLP] make
35350 c93c1e00  103 35349   758 0004000 [SLPQ wait 0xc93c1e00][SLP] sh
35349 c93c1a00  103 35348   758 0004000 [SLPQ select 0xc075cb24][SLP] make
35348 c93bec00  103 33419   758 0004000 [SLPQ wait 0xc93bec00][SLP] sh
34347 c8d33800  103 34346   758 0004000 [SLPQ wait 0xc8d33800][SLP] sh
34346 c3b3c200  103 33426   758 0004000 [SLPQ select 0xc075cb24][SLP] make
33426 c6569000  103 33419   758 0004000 [SLPQ wait 0xc6569000][SLP] sh
33419 c38c3a00  103 18262   758 0004000 [SLPQ select 0xc075cb24][SLP] make
32618 c8ebe400 1021   702 32618 0004002 [SLPQ select 0xc075cb24][SLP] ssh
18262 c38c4a00  103 18255   758 0004000 [SLPQ wait 0xc38c4a00][SLP] sh
18255 c8ec0000  103 18254   758 0004000 [SLPQ select 0xc075cb24][SLP] make
18254 c8ec0a00  103   894   758 0004000 [SLPQ wait 0xc8ec0a00][SLP] sh
  894 c656a200  103   892   758 0004000 [SLPQ select 0xc075cb24][SLP] make
  892 c3939c00  103   811   758 0004000 [SLPQ wait 0xc3939c00][SLP] sh
  811 c656a000  103   810   758 0004000 [SLPQ select 0xc075cb24][SLP] make
  810 c6569e00  103   760   758 0004000 [SLPQ wait 0xc6569e00][SLP] lockf
  760 c3939e00  103   759   758 0004000 [SLPQ wait 0xc3939e00][SLP] sh
  759 c3b37800  103   758   758 0004000 [SLPQ wait 0xc3b37800][SLP] lockf
  758 c3b37200  103   757   758 0004000 [SLPQ pause 0xc3b37234][SLP] csh
  757 c393ce00    0   750   750 0004100 [SLPQ wait 0xc393ce00][SLP] su
  750 c3ac2a00    0   747   750 0004000 [SLPQ wait 0xc3ac2a00][SLP] sh
  747 c38c4c00    0   588   588 0000000 [SLPQ piperd 0xc3941480][SLP] cron
  702 c3b37400 1021   701   702 0004002 [SLPQ pause 0xc3b37434][SLP] zsh
  701 c3abe800 1021   699   699 0000100 [SLPQ select 0xc075cb24][SLP] sshd
  699 c3b37a00    0   566   699 0004100 [SLPQ sbwait 0xc3b13334][SLP] sshd
  698 c393c600    0     1   698 0004002 [SLPQ ttyin 0xc372d410][SLP] getty
  697 c3b3c000    0     1   697 0004002 [SLPQ ttyin 0xc3744010][SLP] getty
...
db> trace 35659
Tracing pid 35659 tid 100126 td 0xc359da80
sched_switch(c359da80,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(d7775df0,e6a72968,c050da15,d7775df0,0) at sleepq_switch+0xe0
sleepq_wait(d7775df0,0,0,c396329c,b5) at sleepq_wait+0x30
msleep(d7775df0,c070eb90,50,c06aeed5,0) at msleep+0x311
acquire(e6a729c0,120,60000,c359da80,0) at acquire+0x76
lockmgr(d7775df0,202122,c396329c,c359da80) at lockmgr+0x42a
getblk(c3963220,a0cee0,0,4000,0) at getblk+0x12a
breadn(c3963220,a0cee0,0,4000,0) at breadn+0x31
bread(c3963220,a0cee0,0,4000,0) at bread+0x20
ffs_update(c976c110,0,c976c110,c387dc00,1) at ffs_update+0x228
ufs_inactive(e6a72b10,c976c18c,c976c110,e6a72b28,c055b84a) at ufs_inactive+0x16c
VOP_INACTIVE_APV(c06f7380,e6a72b10) at VOP_INACTIVE_APV+0x9b
vinactive(c976c110,c359da80) at vinactive+0x8a
vput(c976c110,c387dc00,c0704720,c976c110,3) at vput+0x160
vn_close(c976c110,3,c3a21480,c359da80,e6a72bd8) at vn_close+0x96
vn_closefile(c3938750,c359da80) at vn_closefile+0xca
fdrop_locked(c3938750,c359da80,c34a0fac,0,c06a41df) at fdrop_locked+0x88
fdrop(c3938750,c359da80,6af,c0716380,0) at fdrop+0x24
closef(c3938750,c359da80,0,0,3) at closef+0x35f
db> trace 35655
Tracing pid 35655 tid 100118 td 0xc3b38480
sched_switch(c3b38480,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(d766c638,ecf9eacc,c050da15,d766c638,0) at sleepq_switch+0xe0
sleepq_wait(d766c638,0,0,c06ae919,e52) at sleepq_wait+0x30
msleep(d766c638,c075d140,4c,c06af047,0) at msleep+0x311
bwait(d766c638,4c,c06af047) at bwait+0x47
bufwait(d766c638,1,0,0,ecf9ebb8) at bufwait+0x1a
breadn(c82c4220,0,0,1000,0) at breadn+0x266
bread(c82c4220,0,0,1000,0) at bread+0x20
ffs_read(ecf9ec04,c39381f8,c82c4220,ecf9ec50,c0565a6a) at ffs_read+0x23f
VOP_READ_APV(c06f7380,ecf9ec04) at VOP_READ_APV+0x9b
vn_read(c39381f8,ecf9ec78,c3a21480,0,c3b38480) at vn_read+0x196
dofileread(c3b38480,c39381f8,3,845e000,a9e) at dofileread+0xad
read(c3b38480,ecf9ed04,3,9,202) at read+0x3b
syscall(3b,3b,3b,845e000,a9e) at syscall+0x22f
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (3, FreeBSD ELF32, read), eip = 0x82d05d3, esp = 0xbfbfe60c, ebp = 0xbfbfe638 ---
db> trace 35648
Tracing pid 35648 tid 100167 td 0xc8d32300
sched_switch(c8d32300,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c8fd1480,0,c8d32300,ed04bb68,c050d9d9) at sleepq_switch+0xe0
sleepq_timedwait_sig(c8fd1480,0,0,100,c06a45e3) at sleepq_timedwait_sig+0xd
msleep(c8fd1480,c8fd1480,158,c06a46bf,1388) at msleep+0x2d5
kqueue_scan(c8fd1480,1,ed04bcc8,ed04bcc0,ed04bbf4) at kqueue_scan+0x221
kern_kevent(c8d32300,5,0,1,ed04bcc8) at kern_kevent+0x151
kevent(c8d32300,ed04bd04,6,10,292) at kevent+0x55
syscall(3b,3b,3b,805f000,bfbfe3b0) at syscall+0x22f
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (363, FreeBSD ELF32, kevent), eip = 0x280c9503, esp = 0xbfbfe31c, ebp = 0xbfbfe458 ---
db> trace 35647
Tracing pid 35647 tid 100136 td 0xc3b38180
sched_switch(c3b38180,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
sleepq_switch(c8ffc880,0,c3b38180,ecf98b68,c050d9d9) at sleepq_switch+0xe0
sleepq_timedwait_sig(c8ffc880,0,0,100,c06a45e3) at sleepq_timedwait_sig+0xd
msleep(c8ffc880,c8ffc880,158,c06a46bf,1389) at msleep+0x2d5
kqueue_scan(c8ffc880,1,ecf98cc8,ecf98cc0,ecf98bf4) at kqueue_scan+0x221
kern_kevent(c3b38180,5,0,1,ecf98cc8) at kern_kevent+0x151
kevent(c3b38180,ecf98d04,6,e,292) at kevent+0x55
syscall(3b,3b,3b,805f000,bfbfe3b0) at syscall+0x22f
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (363, FreeBSD ELF32, kevent), eip = 0x280c9503, esp = 0xbfbfe31c, ebp = 0xbfbfe458 ---


-- 
Jun Kuriyama <kuriyama_at_imgsrc.co.jp> // IMG SRC, Inc.
             <kuriyama_at_FreeBSD.org> // FreeBSD Project
Received on Fri Jun 17 2005 - 01:45:13 UTC

This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:38:36 UTC