On Feb 18, 2005, at 15:49, Robert Watson wrote: > > I upgraded a box to a recent 6.x this morning, and ran into the > following > apparently deadlock while backing up a partitionusing dump -L. It > appears > to have gotten stuck during mksnap_ffs. Some interpretation below, but > the high level summary is that it looks to me like we have a lock order > issue between the snapshot write lock for the file system and vnode > locks, > which occurs because vrele() calls VOP_INACTIVE with the vnode lock > held, > and ufs_inactive() then proceeds to get a write reference to the file > system. It could be a fix is to grab the write reference to the file > system in vrele() before the vnode lock, to aovid grabbing the write > reference in ufs_inactive(). My 5.2.1 and RELENG_5 NFS servers all hang up every time I try to use dump -L too. However, background fsck, with some exceptions allways works. One time a system came up after a crash just before the nightly periodic scripts where to be run (lots of finds running etc), and that made it deadlock in a similar way. But except for this incident I have made good use of background fsck countless times. I have one of the NFS servers on serial console, so I decided to retry a dump -L now with RELENG_5, since I am doing some changes to our backup routines anyway. I completely forgot do call doadump though, so I probably lost some valuable information :-( Here is what I extracted: db> show lockedvnods Locked vnodes 0xc418ca50: tag ufs, type VDIR, usecount 3, writecount 0, refcount 1, flags (VV_OBJBUF), lock type ufs: EXCL (count 1) by thread 0xc8e1c320 (pid 89338) ino 3, on dev da1s1c (4, 18) 0xc66f1528: tag ufs, type VREG, usecount 2, writecount 0, refcount 4103, lock type ufs: EXCL (count 1) by thread 0xc3e4a7d0 (pid 88381) with 1 pending ino 274, on dev da1s1c (4, 18) 0xc799a630: tag ufs, type VREG, usecount 1, writecount 0, refcount 147, flags (VV_OBJBUF), lock type ufs: EXCL (count 1) by thread 0xc3914af0 (pid 418) with 2 pending ino 41430101, on dev da1s1c (4, 18) db> ps pid proc uarea uid ppid pgrp flag stat wmesg wchan cmd 89351 c3def710 ee183000 0 5842 89351 0004002 [SLPQ ttyin 0xc90a9c10][SLP] bash 89338 c8e1954c ee2e2000 0 89255 89338 0004002 [SLPQ ufs 0xc66f15d4][SLP] ls 89255 c3defe20 ee187000 0 5842 89255 0004002 [SLPQ wait 0xc3defe20][SLP] bash 88630 c3dea388 ee11c000 0 85239 88630 0004002 [SLPQ select 0xc08efcc4][SLP] top 88381 c3e47e20 ee229000 0 88380 88298 0004002 [SLPQ ufs 0xc799a6dc][SLP] mksnap_ffs 88380 c3e4c000 ee233000 0 88379 88298 0004002 [SLPQ wait 0xc3e4c000][SLP] sh 88379 c398b388 ee061000 0 88298 88298 0004002 [SLPQ wait 0xc398b388][SLP] dump 88298 c3bd5000 ee078000 0 5843 88298 0004002 [SLPQ wait 0xc3bd5000][SLP] sh 85239 c3e4c54c ee236000 0 5842 85239 0004002 [SLPQ wait 0xc3e4c54c][SLP] bash 83213 c3e47a98 ee227000 0 83201 83213 0004002 [SLPQ ttyin 0xc97a0c10][SLP] bash 83201 c3e4c388 ee235000 0 496 83201 0000100 [SLPQ select 0xc08efcc4][SLP] sshd 82285 c3e47c5c ee228000 0 81759 82285 0004002 [SLPQ pause 0xc3e47c94][SLP] screen 81759 c3df88d4 ee1df000 0 81752 81759 0004002 [SLPQ wait 0xc3df88d4][SLP] bash 81752 c3df8388 ee1dc000 0 496 81752 0000100 [SLPQ select 0xc08efcc4][SLP] sshd 5843 c3def8d4 ee184000 0 5842 5843 0004002 [SLPQ wait 0xc3def8d4][SLP] bash 5842 c3e47388 ee1e5000 0 1 5842 0000000 [SLPQ select 0xc08efcc4][SLP] screen 13643 c3df8a98 ee1e0000 0 1 13643 0004002 [SLPQ ttyin 0xc38fe210][SLP] getty 920 c3deaa98 ee17c000 0 564 920 0004002 [SLPQ ttyin 0xc3619e10][SLP] bash 569 c3bda54c ee0de000 0 1 569 0004002 [SLPQ ttyin 0xc3680c10][SLP] getty 568 c3915c5c ebe2a000 0 1 568 0004002 [SLPQ ttyin 0xc3680e10][SLP] getty 567 c3bdaa98 ee0e1000 0 1 567 0004002 [SLPQ ttyin 0xc3681010][SLP] getty 566 c3bd58d4 ee0d7000 0 1 566 0004002 [SLPQ ttyin 0xc3681210][SLP] getty 565 c398b000 ee05f000 0 1 565 0004002 [SLPQ ttyin 0xc3634a10][SLP] getty 564 c398be20 ee067000 0 1 564 0004102 [SLPQ wait 0xc398be20][SLP] login 563 c3986e20 ee05e000 0 1 563 0004002 [SLPQ ttyin 0xc3619810][SLP] login 551 c39121c4 ebdfd000 0 1 551 0000000 [SLPQ select 0xc08efcc4][SLP] inetd 526 c3bd5388 ee098000 0 1 114 0000002 [SLPQ nanslp 0xc08ea56c][SLP] perl 518 c3bd5a98 ee0d8000 0 1 518 0000000 [SLPQ nanslp 0xc08ea56c][SLP] cron 506 c3bda388 ee0dd000 25 1 506 0000100 [SLPQ pause 0xc3bda3c0][SLP] sendmail 502 c3bd51c4 ee07f000 0 1 502 0000100 [SLPQ select 0xc08efcc4][SLP] sendmail 496 c398b1c4 ee060000 0 1 496 0000100 [SLPQ select 0xc08efcc4][SLP] sshd 485 c3bd5c5c ee0d9000 1 1 485 0000100 [SLPQ sbwait 0xc3bb3974][SLP] rwhod 474 c39861c4 ee057000 0 1 474 0000000 [SLPQ select 0xc08efcc4][SLP] ntpd 450 c398654c ee059000 0 1 450 0000000 [SLPQ select 0xc08efcc4][SLP] usbd 435 c3912710 ebe00000 0 429 429 0000100 [SLPQ select 0xc08efcc4][SLP] rpc.lockd 429 c398b54c ee062000 0 1 429 0000000 [SLPQ select 0xc08efcc4][SLP] rpc.lockd 424 c3912000 ebdfc000 0 1 424 0000000 [SLPQ select 0xc08efcc4][SLP] rpc.statd 418 c3912a98 ebe20000 0 415 415 0000000 [SLPQ suspfs 0xc38ef46c][SLP] nfsd 417 c398b8d4 ee064000 0 415 415 0000000 [SLPQ suspfs 0xc38ef46c][SLP] nfsd 416 c391254c ebdff000 0 415 415 0000000 [SLPQ suspfs 0xc38ef46c][SLP] nfsd 415 c39868d4 ee05b000 0 1 415 0000000 [SLPQ ufs 0xc799a6dc][SLP] nfsd 413 c39128d4 ebe01000 0 1 413 0000000 [SLPQ select 0xc08efcc4][SLP] mountd 344 c3915e20 ebe2b000 0 1 344 0000000 [SLPQ select 0xc08efcc4][SLP] rpcbind 325 c35f1e20 ebd27000 0 1 325 0000000 [SLPQ select 0xc08efcc4][SLP] syslogd 305 c3986388 ee058000 0 1 305 0000000 [SLPQ select 0xc08efcc4][SLP] devd 113 c3915000 ebe23000 0 0 0 0000204 [SLPQ - 0xebe10d18][SLP] schedcpu 112 c39151c4 ebe24000 0 0 0 0000204 [SLPQ - 0xc08f76ec][SLP] nfsiod 3 111 c3915388 ebe25000 0 0 0 0000204 [SLPQ - 0xc08f76e8][SLP] nfsiod 2 110 c391554c ebe26000 0 0 0 0000204 [SLPQ - 0xc08f76e4][SLP] nfsiod 1 109 c3915710 ebe27000 0 0 0 0000204 [SLPQ - 0xc08f76e0][SLP] nfsiod 0 108 c39158d4 ebe28000 0 0 0 0000204 [SLPQ hptrdy 0xc078fda4][SLP] hpt_wt 107 c353c1c4 ebcdb000 0 0 0 0000204 [SLPQ vlruwt 0xc353c1c4][SLP] vnlru 106 c353c388 ebcdc000 0 0 0 0000204 [SLPQ syncer 0xc08ea2ec][SLP] syncer 105 c353c54c ebcdd000 0 0 0 0000204 [SLPQ psleep 0xc08f028c][SLP] bufdaemon 104 c353c710 ebcde000 0 0 0 000020c [SLPQ pgzero 0xc08fe054][SLP] pagezero 103 c353c8d4 ebcdf000 0 0 0 0000204 [SLPQ psleep 0xc08fe0a8][SLP] vmdaemon 102 c353ca98 ebce0000 0 0 0 0000204 [SLPQ psleep 0xc08fe064][SLP] pagedaemon 101 c353cc5c ebce1000 0 0 0 0000204 [IWAIT] swi0: sio 100 c353ce20 ebd1e000 0 0 0 0000204 [SLPQ - 0xc35f303c][SLP] fdc0 99 c35f1000 ebd1f000 0 0 0 0000204 [SLPQ usbevt 0xc362b210][SLP] usb2 98 c35f11c4 ebd20000 0 0 0 0000204 [SLPQ usbevt 0xc3627210][SLP] usb1 97 c35f1388 ebd21000 0 0 0 0000204 [SLPQ usbtsk 0xc08e2138][SLP] usbtask 96 c35f154c ebd22000 0 0 0 0000204 [SLPQ usbevt 0xc3611210][SLP] usb0 95 c35f1710 ebd23000 0 0 0 0000204 [SLPQ idle 0xc361b000][SLP] aic_recovery1 94 c35f18d4 ebd24000 0 0 0 0000204 [SLPQ idle 0xc3617000][SLP] aic_recovery0 9 c35f1a98 ebd25000 0 0 0 0000204 [SLPQ actask 0xc0a34a2c][SLP] acpi_task2 8 c35f1c5c ebd26000 0 0 0 0000204 [SLPQ actask 0xc0a34a2c][SLP] acpi_task1 7 c351f54c ebcad000 0 0 0 0000204 [SLPQ actask 0xc0a34a2c][SLP] acpi_task0 93 c351f710 ebcae000 0 0 0 0000204 [IWAIT] swi6:+ 6 c351f8d4 ebcaf000 0 0 0 0000204 [SLPQ - 0xc358fbc0][SLP] thread taskq 92 c351fa98 ebcb0000 0 0 0 0000204 [IWAIT] swi6: acpitaskq 91 c351fc5c ebcb1000 0 0 0 0000204 [IWAIT] swi6:+ 90 c351fe20 ebcb2000 0 0 0 0000204 [IWAIT] swi6: task queue 5 c353a000 ebcb3000 0 0 0 0000204 [SLPQ - 0xc358fd80][SLP] kqueue taskq 89 c353a1c4 ebcd2000 0 0 0 0000204 [IWAIT] swi3: cambio 88 c353a388 ebcd3000 0 0 0 0000204 [IWAIT] swi2: camnet 87 c353a54c ebcd4000 0 0 0 0000204 [SLPQ - 0xc08dfe80][SLP] yarrow 4 c353a710 ebcd5000 0 0 0 0000204 [SLPQ - 0xc08e4b28][SLP] g_down 3 c353a8d4 ebcd6000 0 0 0 0000204 [SLPQ - 0xc08e4b24][SLP] g_up 2 c353aa98 ebcd7000 0 0 0 0000204 [SLPQ - 0xc08e4b1c][SLP] g_event 86 c353ac5c ebcd8000 0 0 0 0000204 [IWAIT] swi4: vm 85 c353ae20 ebcd9000 0 0 0 000020c [IWAIT] swi5: clock sio 84 c353c000 ebcda000 0 0 0 0000204 [IWAIT] swi1: net 83 c350da98 ebc62000 0 0 0 0000204 [IWAIT] irq0: clk 82 c350dc5c ebc63000 0 0 0 0000204 [IWAIT] irq71: 81 c350de20 ebc64000 0 0 0 0000204 [IWAIT] irq70: 80 c351b000 ebc65000 0 0 0 0000204 [IWAIT] irq69: 79 c351b1c4 ebc66000 0 0 0 0000204 [IWAIT] irq68: 78 c351b388 ebc67000 0 0 0 0000204 [IWAIT] irq67: 77 c351b54c ebca4000 0 0 0 0000204 [IWAIT] irq66: 76 c351b710 ebca5000 0 0 0 0000204 [IWAIT] irq65: 75 c351b8d4 ebca6000 0 0 0 0000204 [IWAIT] irq64: 74 c351ba98 ebca7000 0 0 0 0000204 [IWAIT] irq63: 73 c351bc5c ebca8000 0 0 0 0000204 [IWAIT] irq62: 72 c351be20 ebca9000 0 0 0 0000204 [IWAIT] irq61: 71 c351f000 ebcaa000 0 0 0 0000204 [IWAIT] irq60: 70 c351f1c4 ebcab000 0 0 0 0000204 [IWAIT] irq59: em1 69 c351f388 ebcac000 0 0 0 0000204 [IWAIT] irq58: em0 68 c34fc1c4 ebc18000 0 0 0 0000204 [IWAIT] irq57: 67 c34fc388 ebc19000 0 0 0 0000204 [IWAIT] irq56: 66 c34fc54c ebc1a000 0 0 0 0000204 [IWAIT] irq55: 65 c34fc710 ebc1b000 0 0 0 0000204 [IWAIT] irq54: 64 c34fc8d4 ebc1c000 0 0 0 0000204 [IWAIT] irq53: 63 c34fca98 ebc59000 0 0 0 0000204 [IWAIT] irq52: iir0 62 c34fcc5c ebc5a000 0 0 0 0000204 [IWAIT] irq51: 61 c34fce20 ebc5b000 0 0 0 0000204 [IWAIT] irq50: 60 c350d000 ebc5c000 0 0 0 0000204 [IWAIT] irq49: 59 c350d1c4 ebc5d000 0 0 0 0000204 [IWAIT] irq48: 58 c350d388 ebc5e000 0 0 0 0000204 [IWAIT] irq47: 57 c350d54c ebc5f000 0 0 0 0000204 [IWAIT] irq46: 56 c350d710 ebc60000 0 0 0 0000204 [IWAIT] irq45: 55 c350d8d4 ebc61000 0 0 0 0000204 [IWAIT] irq44: 54 c34e6a98 ebbed000 0 0 0 0000204 [IWAIT] irq43: 53 c34e6c5c ebbee000 0 0 0 0000204 [IWAIT] irq42: 52 c34e6e20 ebbef000 0 0 0 0000204 [IWAIT] irq41: 51 c34f8000 ebbf0000 0 0 0 0000204 [IWAIT] irq40: 50 c34f81c4 ebc0f000 0 0 0 0000204 [IWAIT] irq39: 49 c34f8388 ebc10000 0 0 0 0000204 [IWAIT] irq38: 48 c34f854c ebc11000 0 0 0 0000204 [IWAIT] irq37: 47 c34f8710 ebc12000 0 0 0 0000204 [IWAIT] irq36: 46 c34f88d4 ebc13000 0 0 0 0000204 [IWAIT] irq35: 45 c34f8a98 ebc14000 0 0 0 0000204 [IWAIT] irq34: 44 c34f8c5c ebc15000 0 0 0 0000204 [IWAIT] irq33: 43 c34f8e20 ebc16000 0 0 0 0000204 [IWAIT] irq32: 42 c34fc000 ebc17000 0 0 0 0000204 [IWAIT] irq31: ahd1 41 c34dc54c ebbc3000 0 0 0 0000204 [IWAIT] irq30: ahd0 40 c34dc710 ebbc4000 0 0 0 0000204 [IWAIT] irq29: 39 c34dc8d4 ebbc5000 0 0 0 0000204 [IWAIT] irq28: 38 c34dca98 ebbe4000 0 0 0 0000204 [IWAIT] irq27: 37 c34dcc5c ebbe5000 0 0 0 0000204 [IWAIT] irq26: 36 c34dce20 ebbe6000 0 0 0 0000204 [IWAIT] irq25: 35 c34e6000 ebbe7000 0 0 0 0000204 [IWAIT] irq24: 34 c34e61c4 ebbe8000 0 0 0 0000204 [IWAIT] irq23: 33 c34e6388 ebbe9000 0 0 0 0000204 [IWAIT] irq22: 32 c34e654c ebbea000 0 0 0 0000204 [IWAIT] irq21: 31 c34e6710 ebbeb000 0 0 0 0000204 [IWAIT] irq20: 30 c34e68d4 ebbec000 0 0 0 0000204 [IWAIT] irq19: uhci1 29 c348c1c4 ea178000 0 0 0 0000204 [IWAIT] irq18: uhci2 28 c348c388 ea179000 0 0 0 0000204 [IWAIT] irq17: 27 c348c54c ea198000 0 0 0 0000204 [IWAIT] irq16: uhci0 26 c348c710 ea199000 0 0 0 0000204 [IWAIT] irq15: ata1 25 c348c8d4 ea19a000 0 0 0 0000204 [IWAIT] irq14: ata0 24 c348ca98 ea19b000 0 0 0 0000204 [IWAIT] irq13: 23 c348cc5c ea19c000 0 0 0 0000204 [IWAIT] irq12: 22 c348ce20 ea19d000 0 0 0 0000204 [IWAIT] irq11: 21 c34dc000 ebbc0000 0 0 0 0000204 [IWAIT] irq10: 20 c34dc1c4 ebbc1000 0 0 0 0000204 [IWAIT] irq9: acpi0 19 c34dc388 ebbc2000 0 0 0 0000204 [IWAIT] irq8: rtc 18 c3482000 ea132000 0 0 0 0000204 [IWAIT] irq7: ppc0 17 c34821c4 ea16f000 0 0 0 0000204 [IWAIT] irq6: fdc0 16 c3482388 ea170000 0 0 0 0000204 [IWAIT] irq5: 15 c348254c ea171000 0 0 0 0000204 [IWAIT] irq4: sio0 14 c3482710 ea172000 0 0 0 0000204 [IWAIT] irq3: sio1 13 c34828d4 ea173000 0 0 0 0000204 [IWAIT] irq1: atkbd0 12 c3482a98 ea174000 0 0 0 000020c [CPU 0] idle: cpu0 11 c3482c5c ea175000 0 0 0 000020c [CPU 1] idle: cpu1 1 c3482e20 ea176000 0 0 1 0004200 [SLPQ wait 0xc3482e20][SLP] init 10 c348c000 ea177000 0 0 0 0000204 [SLPQ ktrace 0xc08e8518][SLP] ktrace 0 c08e4c20 c0c1f000 0 0 0 0000200 [SLPQ sched 0xc08e4c20][SLP] swapper db> trace 88381 Tracing pid 88381 tid 100235 td 0xc3e4a7d0 sched_switch(c3e4a7d0,0,1) at sched_switch+0x143 mi_switch(1,0,c3e4a7d0,1,c3e4a7d0) at mi_switch+0x1ba sleepq_switch(c799a6dc) at sleepq_switch+0x133 sleepq_wait(c799a6dc,0,f3,0,c3c774a0) at sleepq_wait+0xb msleep(c799a6dc,c08e93c4,50,c08303e7,0) at msleep+0x322 acquire(ee21a7c4,1000040,600,c3e4a7d0,0) at acquire+0xa2 lockmgr(c799a6dc,1010002,c799a630,c3e4a7d0,10002) at lockmgr+0x366 ufs_lock(ee21a7f0,ee21a80c,c0674b20,ee21a7f0,c08ca8e0) at ufs_lock+0x3c ufs_vnoperate(ee21a7f0) at ufs_vnoperate+0x13 vn_lock(c799a630,10002,c3e4a7d0) at vn_lock+0x104 ffs_snapshot(c38ef400,bfbfe9eb,c3911a00,ee21aab0,0) at ffs_snapshot+0x11d2 ffs_omount(c38ef400,c9a96480,bfbfe248,c3e4a7d0,0) at ffs_omount+0x52e vfs_domount(c3e4a7d0,c8cc0990,c9a96480,1211118,bfbfe248) at vfs_domount+0x7b8 mount(c3e4a7d0,ee21ad14,4,0,206) at mount+0x6c syscall(2f,2f,2f,bfbfe498,bfbfe9e1) at syscall+0x283 Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (21, FreeBSD ELF32, mount), eip = 0x280c1473, esp = 0xbfbfe1bc, ebp = 0xbfbfe8b0 --- db> trace 415 Tracing pid 415 tid 100143 td 0xc39884b0 sched_switch(c39884b0,0,1) at sched_switch+0x143 mi_switch(1,0,c39884b0,1,c39884b0) at mi_switch+0x1ba sleepq_switch(c799a6dc) at sleepq_switch+0x133 sleepq_wait(c799a6dc,0,0,0,0) at sleepq_wait+0xb msleep(c799a6dc,c08e93c4,50,c08303e7,0) at msleep+0x322 acquire(ee0448ec,1000040,600,c39884b0,0) at acquire+0xa2 lockmgr(c799a6dc,1010002,c799a630,c39884b0,10002) at lockmgr+0x366 ufs_lock(ee044918,ee044934,c0674b20,ee044918,c08ca8e0) at ufs_lock+0x3c ufs_vnoperate(ee044918) at ufs_vnoperate+0x13 vn_lock(c799a630,10002,c39884b0,2782c55,4) at vn_lock+0x104 vget(c799a630,10002,c39884b0,c799a630,c39884b0) at vget+0xfd ufs_ihashget(c393dd00,2782c55,2,ee044a0c,1010002) at ufs_ihashget+0xfe ffs_vget(c38ef400,2782c55,2,ee044a0c) at ffs_vget+0x31 ufs_fhtovp(c38ef400,ee044bf8,ee044ae4,c38ef400,ee044a70) at ufs_fhtovp+0x22 ffs_fhtovp(c38ef400,ee044bf8,ee044ae4,c39884b0,100) at ffs_fhtovp+0x3a nfsrv_fhtovp(ee044bf0,1,ee044ae4,ca071380,c3bb1780) at nfsrv_fhtovp+0xf5 nfsrv_read(ca071300,c3bb1780,c39884b0,ee044ca8,ee044ca8) at nfsrv_read+0x25d nfssvc_nfsd(c39884b0,0,c39884b0,ee044ce8,c39868d4) at nfssvc_nfsd+0x406 nfssvc(c39884b0,ee044d14,2,1,296) at nfssvc+0x1bc syscall(2f,2f,2f,bfbfedf4,8052180) at syscall+0x283 Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (155, FreeBSD ELF32, nfssvc), eip = 0x280c2ab3, esp = 0xbfbfeb1c, ebp = 0xbfbfeb38 --- db> trace 416 Tracing pid 416 tid 100098 td 0xc353bc80 sched_switch(c353bc80,0,1) at sched_switch+0x143 mi_switch(1,0,c353bc80,1,c353bc80) at mi_switch+0x1ba sleepq_switch(c38ef46c) at sleepq_switch+0x133 sleepq_wait(c38ef46c,0,801031d,0,0) at sleepq_wait+0xb msleep(c38ef46c,0,9f,c0831fcf,0) at msleep+0x322 vn_start_write(0,ebcceabc,1) at vn_start_write+0x8b nfsrv_write(ca071900,c3bb1780,c353bc80,ebcceca8,ebcceca0) at nfsrv_write+0x1b4 nfssvc_nfsd(c353bc80,0,c353bc80,ebccece8,c391254c) at nfssvc_nfsd+0x406 nfssvc(c353bc80,ebcced14,2,0,296) at nfssvc+0x1bc syscall(2f,2f,2f,bfbfeec4,3) at syscall+0x283 Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (155, FreeBSD ELF32, nfssvc), eip = 0x280c2ab3, esp = 0xbfbfeb1c, ebp = 0xbfbfeb38 --- db> trace 417 Tracing pid 417 tid 100134 td 0xc3987640 sched_switch(c3987640,0,1) at sched_switch+0x143 mi_switch(1,0,c3987640,1,c3987640) at mi_switch+0x1ba sleepq_switch(c38ef46c) at sleepq_switch+0x133 sleepq_wait(c38ef46c,0,3a,0,c364bba0) at sleepq_wait+0xb msleep(c38ef46c,0,9f,c0831fcf,0) at msleep+0x322 vn_start_write(0,ee029abc,1) at vn_start_write+0x8b nfsrv_write(ca2e2000,c3bb1780,c3987640,ee029ca8,ee029ca0) at nfsrv_write+0x1b4 nfssvc_nfsd(c3987640,0,c3987640,ee029ce8,c398b8d4) at nfssvc_nfsd+0x406 nfssvc(c3987640,ee029d14,2,0,296) at nfssvc+0x1bc syscall(2f,2f,2f,bfbfeec4,3) at syscall+0x283 Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (155, FreeBSD ELF32, nfssvc), eip = 0x280c2ab3, esp = 0xbfbfeb1c, ebp = 0xbfbfeb38 --- db> trace 418 Tracing pid 418 tid 100127 td 0xc3914af0 sched_switch(c3914af0,0,1) at sched_switch+0x143 mi_switch(1,0,c3914af0,1,c3914af0) at mi_switch+0x1ba sleepq_switch(c38ef46c) at sleepq_switch+0x133 sleepq_wait(c38ef46c,0,c08ca3e0,0,c364bba0) at sleepq_wait+0xb msleep(c38ef46c,0,9f,c0831fcf,0) at msleep+0x322 vn_write_suspend_wait(c799a630,c38ef400,1,0,dc59b08c) at vn_write_suspend_wait+0x89 ufs_inactive(ebe19a74,ebe19a8c,c0669ae7,ebe19a74,c08ca860) at ufs_inactive+0x1ba ufs_vnoperate(ebe19a74) at ufs_vnoperate+0x13 vput(c799a630) at vput+0xfb nfsrv_read(ca10f000,c3bb1780,c3914af0,ebe19ca8,ebe19ca0) at nfsrv_read+0xcfa nfssvc_nfsd(c3914af0,0,c3914af0,ebe19ce8,c3912a98) at nfssvc_nfsd+0x406 nfssvc(c3914af0,ebe19d14,2,0,296) at nfssvc+0x1bc syscall(2f,2f,2f,bfbfeec4,3) at syscall+0x283 Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (155, FreeBSD ELF32, nfssvc), eip = 0x280c2ab3, esp = 0xbfbfeb1c, ebp = 0xbfbfeb38 --- db> Regards, Frode Nordahl > Robert N M Watson > > www:~> more backup.csh > #!/bin/csh > > set DATE=`date +"%Y%m%d"` > > dump -L -f - -a -u -0 /home/pnn | gzip -9 > ${DATE}-backup.dump.gz > www:~> df > Filesystem 1K-blocks Used Avail Capacity Mounted on > /dev/twed0s1a 507630 103502 363518 22% / > devfs 1 1 0 100% /dev > /dev/twed0s1f 89359640 370128 81840742 0% /home/pnn > /dev/twed0s1e 16244334 3211132 11733656 21% /usr > /dev/twed0s1d 8122126 1541978 5930378 21% /var > www:~> ./backup.csh > load: 0.00 cmd: mksnap_ffs 2993 [ufs] 0.00u 3.88s 0% 496k > load: 0.00 cmd: mksnap_ffs 2993 [ufs] 0.00u 3.88s 0% 496k > load: 0.00 cmd: mksnap_ffs 2993 [ufs] 0.00u 3.88s 0% 496k > load: 0.00 cmd: mksnap_ffs 2993 [ufs] 0.00u 3.88s 0% 496k > > On the serial console: > > telnet> send break > KDB: enter: Line break on console > [thread pid 11 tid 100004 ] > Stopped at kdb_enter+0x2b: nop > db> show lockedvnods > Locked vnodes > > 0xc13c49b4: tag ufs, type VDIR > usecount 153, writecount 0, refcount 1 mountedhere 0 > flags (VV_ROOT) > v_object 0 > lock type ufs: EXCL (count 1) by thread 0xc1807a10 (pid 3057) with > 152 pend > ing > ino 2, on dev twed0s1f (229, 9) > > 0xc15268a0: tag ufs, type VDIR > usecount 2, writecount 0, refcount 1 mountedhere 0 > flags () > v_object 0 > lock type ufs: EXCL (count 1) by thread 0xc137e000 (pid 3062) > with 1 > pendin > g > ino 3085322, on dev twed0s1f (229, 9) > > 0xc163bbdc: tag ufs, type VDIR > usecount 2, writecount 0, refcount 1 mountedhere 0 > flags () > v_object 0 > lock type ufs: EXCL (count 1) by thread 0xc162dcf0 (pid 3056) > with 1 > pendin > g > ino 3133686, on dev twed0s1f (229, 9) > > 0xc17e933c: tag ufs, type VREG > usecount 1, writecount 0, refcount 0 mountedhere 0 > flags () > v_object 0xc17e7210 > lock type ufs: EXCL (count 1) by thread 0xc1996450 (pid 3045) > with 1 > pendin > g > ino 3134802, on dev twed0s1f (229, 9) > > 0xc18419b4: tag ufs, type VREG > usecount 0, writecount 0, refcount 0 mountedhere 0 > flags () > v_object 0xc1826000 > lock type ufs: EXCL (count 1) by thread 0xc19b2450 (pid 3047) > ino 3132762, on dev twed0s1f (229, 9) > > 0xc1ac933c: tag ufs, type VREG > usecount 1, writecount 0, refcount 865 mountedhere 0 > flags () > v_object 0 > lock type ufs: EXCL (count 1) by thread 0xc162d8a0 (pid 2993) > ino 9891841, on dev twed0s1f (229, 9) > > 0xc19fd9b4: tag ufs, type VREG > usecount 0, writecount 0, refcount 0 mountedhere 0 > flags () > v_object 0xc1a93948 > lock type ufs: EXCL (count 1) by thread 0xc1807b80 (pid 2609) > ino 3197600, on dev twed0s1f (229, 9) > db> ps > pid proc uid ppid pgrp flag stat wmesg wchan cmd > 3296 c1cf93f8 1062 3292 3292 4000000 [SLPQ ufs 0xc13c4a78][SLP] sh > 3295 c1cf91fc 1062 3293 3293 4000000 [SLPQ ufs 0xc13c4a78][SLP] sh > 3293 c1cf9be8 1062 3290 3293 0004000 [SLPQ wait 0xc1cf9be8][SLP] sh > 3292 c1d66000 1062 3289 3292 0004000 [SLPQ wait 0xc1d66000][SLP] sh > 3290 c19b0000 0 390 390 0000000 [SLPQ piperd 0xc1655480][SLP] > cron > 3289 c1d661fc 0 390 390 0000000 [SLPQ piperd 0xc19a3d80][SLP] > cron > 3271 c1d66be8 1062 3268 3268 4000000 [SLPQ ufs 0xc13c4a78][SLP] sh > 3268 c1d69000 1062 3267 3268 0004000 [SLPQ wait 0xc1d69000][SLP] sh > 3267 c1d669ec 0 390 390 0000000 [SLPQ piperd 0xc13aac00][SLP] > cron > 3247 c1cf95f4 1062 3245 3245 4000000 [SLPQ ufs 0xc13c4a78][SLP] sh > 3245 c1cf99ec 1062 3243 3245 0004000 [SLPQ wait 0xc1cf99ec][SLP] sh > 3243 c1cf9de4 0 390 390 0000000 [SLPQ piperd 0xc13aa480][SLP] > cron > 3227 c1d66de4 1062 3222 3222 4000000 [SLPQ ufs 0xc13c4a78][SLP] sh > 3226 c1c8f1fc 1062 3223 3223 4000000 [SLPQ ufs 0xc13c4a78][SLP] sh > 3223 c18063f8 1062 3218 3223 0004000 [SLPQ wait 0xc18063f8][SLP] sh > 3222 c1805000 1062 3217 3222 0004000 [SLPQ wait 0xc1805000][SLP] sh > 3218 c19ae000 0 390 390 0000000 [SLPQ piperd 0xc13a9a80][SLP] > cron > 3217 c18067f0 0 390 390 0000000 [SLPQ piperd 0xc13a9300][SLP] > cron > 3197 c1c8f3f8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3196 c1c8f5f4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3195 c1c8f7f0 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3194 c1c8f9ec 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3193 c1c8fbe8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3192 c1c8fde4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3191 c1cf6000 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3190 c1cf61fc 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3189 c1cf63f8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3188 c1cf65f4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3187 c1cf67f0 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3186 c1cf69ec 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3185 c1cf6be8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3184 c1cf6de4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3183 c1cf9000 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3182 c1c8e1fc 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3181 c1c8e5f4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3180 c1c8e9ec 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3179 c1c1e3f8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3178 c1c1e5f4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3177 c1c1e7f0 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3176 c1c1e9ec 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3175 c1c1ebe8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3174 c1c1ede4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3173 c1c88000 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3172 c1c881fc 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3171 c1c883f8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3170 c1c885f4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3169 c1c887f0 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3168 c1c889ec 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3167 c1c88be8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3166 c1c88de4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3165 c1c8e000 1062 3163 3163 4000000 [SLPQ ufs 0xc13c4a78][SLP] sh > 3163 c1c8e3f8 1062 3161 3163 0004000 [SLPQ wait 0xc1c8e3f8][SLP] sh > 3161 c1c8e7f0 0 390 390 0000000 [SLPQ piperd 0xc19a3000][SLP] > cron > 3159 c1c8ebe8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3158 c1c8ede4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3157 c1c8f000 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3156 c1bc37f0 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3155 c1bc39ec 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3154 c1bc3be8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3153 c1bc3de4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3152 c1c19000 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3151 c1c191fc 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3150 c1c193f8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3149 c1c195f4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3148 c1c197f0 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3147 c1c199ec 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3146 c1c19be8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3145 c1c19de4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3144 c1c1a000 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3143 c1c1a1fc 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3142 c1c1a3f8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3141 c1c1a5f4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3140 c1c1a7f0 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3139 c1c1a9ec 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3138 c1c1abe8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3137 c1c1ade4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3136 c1c1e000 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3135 c1c1e1fc 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3134 c1bc1de4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3133 c1bc1be8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3132 c1b837f0 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3131 c1b839ec 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3130 c1b83be8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3129 c1b83de4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3128 c1bc1000 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3127 c1bc11fc 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3126 c1bc13f8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3125 c1bc15f4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3124 c1bc17f0 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3123 c1bc19ec 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3120 c1bc3000 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3119 c1bc31fc 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3118 c1bc33f8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3117 c1bc35f4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3116 c1abb7f0 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3115 c1abb9ec 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3114 c1abbbe8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3113 c1abbde4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3112 c1b82000 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3111 c1b821fc 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3110 c1b823f8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3109 c1b825f4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3108 c1b827f0 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3107 c1b829ec 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3106 c1b82be8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3105 c1b82de4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3104 c1b83000 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3103 c1b831fc 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3102 c1b833f8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3101 c1b835f4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3100 c19b07f0 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3099 c19b09ec 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3098 c19b0be8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3097 c19b0de4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3096 c1aba000 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3095 c1aba1fc 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3094 c1aba3f8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3093 c1aba5f4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3092 c1aba7f0 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3091 c1aba9ec 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3090 c1ababe8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3089 c1abade4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3088 c1abb000 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3087 c1abb1fc 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3086 c1abb3f8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3085 c16291fc 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3084 c1abb5f4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3083 c1510be8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3082 c162c3f8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3081 c18061fc 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3080 c19957f0 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3079 c18055f4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3078 c19aede4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3077 c19959ec 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3076 c19aebe8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3075 c16297f0 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3074 c16293f8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3073 c19ae5f4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3072 c19ae1fc 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3071 c19951fc 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3070 c19ae7f0 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3069 c12ab7f0 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3068 c1510de4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3067 c1805be8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3066 c1994000 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3065 c19ae9ec 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3064 c137d3f8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3063 c150f000 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3062 c1510000 80 2910 403 0004000 [SLPQ ufs 0xc163bca0][SLP] > download.cgi > 3060 c19953f8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3059 c19947f0 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3058 c16299ec 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3057 c1806be8 80 403 403 0000100 [SLPQ ufs 0xc1526964][SLP] > httpd > 3056 c1994be8 80 403 403 0000100 [SLPQ ufs 0xc17e9400][SLP] > httpd > 3055 c1806000 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3053 c137dde4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3051 c19b05f4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3048 c19941fc 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 3047 c19b03f8 80 403 403 0000100 [SLPQ suspfs 0xc137086c][SLP] > httpd > 3045 c1995be8 80 403 403 0000100 [SLPQ suspfs 0xc137086c][SLP] > httpd > 3036 c19945f4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 2993 c162c1fc 1000 2992 2987 0004102 [SLPQ ufs 0xc13c4a78][SLP] > mksnap_ffs > 2992 c162cbe8 1000 2990 2987 0004002 [SLPQ wait 0xc162cbe8][SLP] sh > 2991 c1995de4 1000 2987 2987 0004002 [SLPQ piperd 0xc13a9780][SLP] > gzip > 2990 c150f5f4 1000 2987 2987 0004002 [SLPQ wait 0xc150f5f4][SLP] > dump > 2987 c1806de4 1000 1762 2987 0004002 [SLPQ pause 0xc1806e18][SLP] > csh > 2910 c1994de4 80 403 403 0000100 [SLPQ wait 0xc1994de4][SLP] > httpd > 2909 c18053f8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 2908 c19ae3f8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 2843 c1995000 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 2839 c19955f4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 2669 c1629000 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 2609 c18069ec 80 403 403 0000100 [SLPQ suspfs 0xc137086c][SLP] > httpd > 2598 c19943f8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 1981 c18051fc 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 1899 c16295f4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 1762 c162cde4 1000 1761 1762 0004002 [SLPQ pause 0xc162ce18][SLP] > tcsh > 1761 c162c000 1000 1759 1759 0000100 [SLPQ select 0xc078a744][SLP] > sshd > 1759 c1629be8 0 368 1759 0004100 [SLPQ sbwait 0xc1429480][SLP] > sshd > 1754 c18065f4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] > httpd > 526 c162c9ec 1062 516 526 0004100 [SLPQ piperd 0xc13a9900][SLP] > sendmail > 525 c162c7f0 1062 1 525 0004000 [SLPQ accept 0xc1429186][SLP] > perl > 516 c18057f0 0 390 390 0000000 [SLPQ piperd 0xc13a9c00][SLP] > cron > 447 c12ab5f4 0 1 447 0004002 [SLPQ ttyin 0xc1318410][SLP] > getty > 446 c15107f0 0 1 446 0004002 [SLPQ ttyin 0xc12f5810][SLP] > getty > 445 c15105f4 0 1 445 0004002 [SLPQ ttyin 0xc12f6810][SLP] > getty > 444 c15101fc 0 1 444 0004002 [SLPQ ttyin 0xc12f6410][SLP] > getty > 443 c150f9ec 0 1 443 0004002 [SLPQ ttyin 0xc12ef410][SLP] > getty > 442 c15103f8 0 1 442 0004002 [SLPQ ttyin 0xc12ef810][SLP] > getty > 441 c150f3f8 0 1 441 0004002 [SLPQ ttyin 0xc12efc10][SLP] > getty > 440 c150f1fc 0 1 440 0004002 [SLPQ ttyin 0xc12f5010][SLP] > getty > 439 c150fde4 0 1 439 0004002 [SLPQ ttyin 0xc12f5410][SLP] > getty > 427 c150fbe8 0 1 427 0000000 [SLPQ select 0xc078a744][SLP] > inetd > 403 c150f7f0 0 1 403 0000000 [SLPQ select 0xc078a744][SLP] > httpd > 390 c12ab9ec 0 1 390 0000000 [SLPQ nanslp 0xc0785dac][SLP] > cron > 378 c137d000 25 1 378 0000100 [SLPQ pause 0xc137d034][SLP] > sendmail > 374 c12abbe8 0 1 374 0000100 [SLPQ select 0xc078a744][SLP] > sendmail > 368 c137d1fc 0 1 368 0000100 [SLPQ select 0xc078a744][SLP] > sshd > 252 c12ab3f8 0 1 252 0000000 [SLPQ select 0xc078a744][SLP] > syslogd > 227 c12abde4 0 1 227 0000000 [SLPQ select 0xc078a744][SLP] > devd > 46 c137d5f4 0 0 0 0000204 [SLPQ - 0xcb30cd14][SLP] > schedcpu > 45 c137d7f0 0 0 0 0000204 [SLPQ - 0xc07923ec][SLP] > nfsiod 3 > 44 c137d9ec 0 0 0 0000204 [SLPQ - 0xc07923e8][SLP] > nfsiod 2 > 43 c137dbe8 0 0 0 0000204 [SLPQ - 0xc07923e4][SLP] > nfsiod 1 > 42 c1243be8 0 0 0 0000204 [SLPQ - 0xc07923e0][SLP] > nfsiod 0 > 41 c1243de4 0 0 0 0000204 [SLPQ vlruwt 0xc1243de4][SLP] > vnlru > 40 c12a8000 0 0 0 0000204 [SLPQ syncer 0xc0785b0c][SLP] > syncer > 39 c12a81fc 0 0 0 0000204 [SLPQ psleep 0xc078ac90][SLP] > bufdaemon > 38 c12a83f8 0 0 0 000020c [SLPQ pgzero 0xc07989e4][SLP] > pagezero > 9 c12a85f4 0 0 0 0000204 [SLPQ psleep 0xc0798534][SLP] > vmdaemon > 8 c12a87f0 0 0 0 0000204 [SLPQ psleep 0xc07984f0][SLP] > pagedaemon > 37 c12a89ec 0 0 0 0000204 [IWAIT] swi0: sio > 7 c12a8be8 0 0 0 0000204 [SLPQ - 0xc120983c][SLP] fdc0 > 36 c12a8de4 0 0 0 0000204 [SLPQ usbtsk 0xc0780814][SLP] > usbtask > 35 c12ab000 0 0 0 0000204 [SLPQ usbevt 0xc12bb210][SLP] > usb0 > 34 c12ab1fc 0 0 0 0000204 [IWAIT] swi5:+ > 6 c12335f4 0 0 0 0000204 [SLPQ - 0xc1263dc0][SLP] thread > taskq > 33 c12337f0 0 0 0 0000204 [IWAIT] swi6:+ > 32 c12339ec 0 0 0 0000204 [IWAIT] swi6: task queue > 5 c1233be8 0 0 0 0000204 [SLPQ - 0xc128e040][SLP] kqueue > taskq > 31 c1233de4 0 0 0 0000204 [IWAIT] swi2: cambio > 30 c1243000 0 0 0 0000204 [SLPQ - 0xc077e7a0][SLP] yarrow > 4 c12431fc 0 0 0 0000204 [SLPQ - 0xc0783148][SLP] g_down > 3 c12433f8 0 0 0 0000204 [SLPQ - 0xc0783144][SLP] g_up > 2 c12435f4 0 0 0 0000204 [SLPQ - 0xc078313c][SLP] > g_event > 29 c12437f0 0 0 0 0000204 [IWAIT] swi3: vm > 28 c12439ec 0 0 0 000020c [IWAIT] swi4: clock sio > 27 c12191fc 0 0 0 0000204 [IWAIT] swi1: net > 26 c12193f8 0 0 0 0000204 [IWAIT] irq15: ata1 > 25 c12195f4 0 0 0 0000204 [IWAIT] irq14: ata0 > 24 c12197f0 0 0 0 0000204 [IWAIT] irq13: > 23 c12199ec 0 0 0 0000204 [IWAIT] irq12: > 22 c1219be8 0 0 0 0000204 [IWAIT] irq11: > 21 c1219de4 0 0 0 0000204 [IWAIT] irq10: > 20 c1233000 0 0 0 0000204 [IWAIT] irq9: twe0 > 19 c12331fc 0 0 0 0000204 [IWAIT] irq8: rtc > 18 c12333f8 0 0 0 0000204 [IWAIT] irq7: ppc0 > 17 c1211000 0 0 0 0000204 [IWAIT] irq6: fdc0 > 16 c12111fc 0 0 0 0000204 [IWAIT] irq5: bge0 > 15 c12113f8 0 0 0 0000204 [IWAIT] irq4: sio0 > 14 c12115f4 0 0 0 0000204 [IWAIT] irq3: uhci0 > 13 c12117f0 0 0 0 0000204 [IWAIT] irq1: atkbd0 > 12 c12119ec 0 0 0 0000204 [IWAIT] irq0: clk > 11 c1211be8 0 0 0 000020c [CPU 0] idle > 1 c1211de4 0 0 1 0004200 [SLPQ wait 0xc1211de4][SLP] > init > 10 c1219000 0 0 0 0000204 [SLPQ ktrace 0xc0783d38][SLP] > ktrace > 0 c07832a0 0 0 0 0000200 [SLPQ sched 0xc07832a0][SLP] > swapper > 517 c162c5f4 1062 516 517 0006000 zomb[INACTIVE] sh > db> show pcpu > cpuid = 0 > curthread = 0xc12125c0: pid 11 "idle" > curpcb = 0xc69b7da0 > fpcurthread = none > idlethread = 0xc12125c0: pid 11 "idle" > APIC ID = 0 > currentldt = 0x28 > > Joe 'sh' process in ufs: > > db> trace 3296 > Tracing pid 3296 tid 100255 td 0xc1cfa730 > sched_switch(c1cfa730,0,1) at sched_switch+0x17f > mi_switch(1,0) at mi_switch+0x264 > sleepq_switch(c13c4a78,cbad385c,c0526901,c13c4a78,0) at > sleepq_switch+0xe0 > sleepq_wait(c13c4a78,0,0,c13c4a78,1010040) at sleepq_wait+0x30 > msleep(c13c4a78,c0784458,50,c06f08fe,0) at msleep+0x2d1 > acquire(cbad38b4,1010040,600,c1cfa730,0) at acquire+0x9a > lockmgr(c13c4a78,1010002,c13c49b4,c1cfa730,cbad38d4) at lockmgr+0x382 > vop_stdlock(cbad3904) at vop_stdlock+0x1b > VOP_LOCK_APV(c076f340,cbad3904,cbad38f4,c06c279e,cbad3904) at > VOP_LOCK_APV+0x7e > ffs_lock(cbad3904) at ffs_lock+0x3b > VOP_LOCK_APV(c076ed60,cbad3904) at VOP_LOCK_APV+0x7e > vn_lock(c13c49b4,10002,c1cfa730,2,c136a800) at vn_lock+0xc2 > vget(c13c49b4,10002,c1cfa730,c1cfa730,c13c4bdc) at vget+0xc9 > ufs_ihashget(c136a800,2,2,cbad3a00,c06e98f4) at ufs_ihashget+0x98 > ffs_vget(c1370800,2,2,cbad3a00) at ffs_vget+0x37 > ufs_root(c1370800,cbad3a3c,c1cfa730,1,1) at ufs_root+0x18 > lookup(cbad3c34,c05197b6,1,c1cfa730,c06ea9fd) at lookup+0x593 > namei(cbad3c34,c05197b6,c1cf9460,1,0) at namei+0x34e > do_execve(c1cfa730,cbad3cc4,0,0,cbad3cc4) at do_execve+0x15d > kern_execve(c1cfa730,cbad3cc4,0,c6295000,c6295000) at kern_execve+0x74 > execve(c1cfa730,cbad3d14,3,0,282) at execve+0x32 > syscall(2f,2f,2f,8067058,806708c) at syscall+0x213 > Xint0x80_syscall() at Xint0x80_syscall+0x1f > --- syscall (59, FreeBSD ELF32, execve), eip = 0x28121d5b, esp = > 0xbfbfec1c, ebp = 0xbfbfec48 --- > > httpd holding a VV_ROOT vnode lock, blocked in ufs: > > db> trace 3057 > Tracing pid 3057 tid 100107 td 0xc1807a10 > sched_switch(c1807a10,0,1) at sched_switch+0x17f > mi_switch(1,0) at mi_switch+0x264 > sleepq_switch(c1526964,cb707994,c0526901,c1526964,0) at > sleepq_switch+0xe0 > sleepq_wait(c1526964,0,0,c1526964,1010040) at sleepq_wait+0x30 > msleep(c1526964,c0785244,50,c06f08fe,0) at msleep+0x2d1 > acquire(cb7079ec,1010040,600,c1807a10,0) at acquire+0x9a > lockmgr(c1526964,1010002,c15268a0,c1807a10,cb707a0c) at lockmgr+0x382 > vop_stdlock(cb707a3c) at vop_stdlock+0x1b > VOP_LOCK_APV(c076f340,cb707a3c,cb707a2c,c06c279e,cb707a3c) at > VOP_LOCK_APV+0x7e > ffs_lock(cb707a3c) at ffs_lock+0x3b > VOP_LOCK_APV(c076ed60,cb707a3c) at VOP_LOCK_APV+0x7e > vn_lock(c15268a0,10002,c1807a10,cb707b24,cb707c38) at vn_lock+0xc2 > vget(c15268a0,2,c1807a10,1b8,c1807a10) at vget+0xc9 > vfs_cache_lookup(cb707b24) at vfs_cache_lookup+0x1bd > VOP_LOOKUP_APV(c076ed60,cb707b24) at VOP_LOOKUP_APV+0x7e > lookup(cb707c10,cb707b70,1,c1807a10,1) at lookup+0x359 > namei(cb707c10,cb707bc4,c0672ebb,c19cbad0,c0703dc4) at namei+0x34e > kern_stat(c1807a10,81541f8,0,cb707c84) at kern_stat+0x35 > stat(c1807a10,cb707d14,2,0,296) at stat+0x1b > syscall(280b002f,2f,bfbf002f,81541f8,8170) at syscall+0x213 > Xint0x80_syscall() at Xint0x80_syscall+0x1f > --- syscall (188, FreeBSD ELF32, stat), eip = 0x2839359b, esp = > 0xbfbfe9dc, ebp = 0xbfbfea78 --- > > download.cgi blocked in ufs holding a directory vnode lock: > > db> trace 3062 > Tracing pid 3062 tid 100050 td 0xc137e000 > sched_switch(c137e000,0,1) at sched_switch+0x17f > mi_switch(1,0) at mi_switch+0x264 > sleepq_switch(c163bca0,cb2e5844,c0526901,c163bca0,0) at > sleepq_switch+0xe0 > sleepq_wait(c163bca0,0,0,c163bca0,1010040) at sleepq_wait+0x30 > msleep(c163bca0,c0784578,50,c06f08fe,0) at msleep+0x2d1 > acquire(cb2e589c,1010040,600,c137e000,0) at acquire+0x9a > lockmgr(c163bca0,1010002,c163bbdc,c137e000,cb2e58bc) at lockmgr+0x382 > vop_stdlock(cb2e58ec) at vop_stdlock+0x1b > VOP_LOCK_APV(c076f340,cb2e58ec,cb2e58dc,c06c279e,cb2e58ec) at > VOP_LOCK_APV+0x7e > ffs_lock(cb2e58ec) at ffs_lock+0x3b > VOP_LOCK_APV(c076ed60,cb2e58ec) at VOP_LOCK_APV+0x7e > vn_lock(c163bbdc,10002,c137e000,cb2e59d4,cb2e5c04) at vn_lock+0xc2 > vget(c163bbdc,2,c137e000,1e0,c137e000) at vget+0xc9 > vfs_cache_lookup(cb2e59d4) at vfs_cache_lookup+0x1bd > VOP_LOOKUP_APV(c076ed60,cb2e59d4) at VOP_LOOKUP_APV+0x7e > lookup(cb2e5bdc,cb2e5a76,1,c137e000,cb2e5a80) at lookup+0x359 > namei(cb2e5bdc,0,d7ec0005,c137e000,c050f370) at namei+0x34e > vn_open_cred(cb2e5bdc,cb2e5cdc,1a4,c19ace00,3) at vn_open_cred+0x27e > vn_open(cb2e5bdc,cb2e5cdc,1a4,3,c07849f8) at vn_open+0x1e > kern_open(c137e000,804a794,0,1,1b6) at kern_open+0xb6 > open(c137e000,cb2e5d14,3,1,292) at open+0x1a > syscall(2f,2f,2f,4,2814e280) at syscall+0x213 > Xint0x80_syscall() at Xint0x80_syscall+0x1f > --- syscall (5, FreeBSD ELF32, open), eip = 0x280ba23b, esp = > 0xbfbfe10c, > ebp = 0xbfbfe138 --- > > httpd holding a directory lock blocked in ufs: > > db> trace 3056 > Tracing pid 3056 tid 100099 td 0xc162dcf0 > sched_switch(c162dcf0,0,1) at sched_switch+0x17f > mi_switch(1,0) at mi_switch+0x264 > sleepq_switch(c17e9400,cb6be994,c0526901,c17e9400,0) at > sleepq_switch+0xe0 > sleepq_wait(c17e9400,0,0,c17e9400,1010040) at sleepq_wait+0x30 > msleep(c17e9400,c0784c38,50,c06f08fe,0) at msleep+0x2d1 > acquire(cb6be9ec,1010040,600,c162dcf0,0) at acquire+0x9a > lockmgr(c17e9400,1010002,c17e933c,c162dcf0,cb6bea0c) at lockmgr+0x382 > vop_stdlock(cb6bea3c) at vop_stdlock+0x1b > VOP_LOCK_APV(c076f340,cb6bea3c,cb6bea2c,c06c279e,cb6bea3c) at > VOP_LOCK_APV+0x7e > ffs_lock(cb6bea3c) at ffs_lock+0x3b > VOP_LOCK_APV(c076ed60,cb6bea3c) at VOP_LOCK_APV+0x7e > vn_lock(c17e933c,10002,c162dcf0,cb6beb24,cb6bec38) at vn_lock+0xc2 > vget(c17e933c,2,c162dcf0,210,c162dcf0) at vget+0xc9 > vfs_cache_lookup(cb6beb24) at vfs_cache_lookup+0x1bd > VOP_LOOKUP_APV(c076ed60,cb6beb24) at VOP_LOOKUP_APV+0x7e > lookup(cb6bec10,c17e9450,1,c162dcf0,75c) at lookup+0x359 > namei(cb6bec10,c05197b6,c0798060,1,c06ea9fd) at namei+0x34e > kern_stat(c162dcf0,8161ad8,0,cb6bec84) at kern_stat+0x35 > stat(c162dcf0,cb6bed14,2,0,292) at stat+0x1b > syscall(280b002f,2f,bfbf002f,8161ad8,8170) at syscall+0x213 > Xint0x80_syscall() at Xint0x80_syscall+0x1f > --- syscall (188, FreeBSD ELF32, stat), eip = 0x2839359b, esp = > 0xbfbfe75c, ebp = 0xbfbfe7f8 --- > > httpd holding a file lock, blocked in suspfs from ufs_inactive: > > db> trace 3045 > Tracing pid 3045 tid 100123 td 0xc1996450 > sched_switch(c1996450,0,1) at sched_switch+0x17f > mi_switch(1,0) at mi_switch+0x264 > sleepq_switch(c137086c,cb6ecb8c,c0526901,c137086c,0) at > sleepq_switch+0xe0 > sleepq_wait(c137086c,0,0,1,c153be38) at sleepq_wait+0x30 > msleep(c137086c,c1370844,29f,c06f2e66,0) at msleep+0x2d1 > vn_write_suspend_wait(c17e933c,c1370800,1,0,c06f2634) at > vn_write_suspend_wait+0xe4 > ufs_inactive(cb6ecc14) at ufs_inactive+0x186 > VOP_INACTIVE_APV(c076ed60,cb6ecc14) at VOP_INACTIVE_APV+0x7e > vrele(c17e933c,c17e7210,0,c07046ce,19d) at vrele+0x10e > vm_object_vndeallocate(c17e7210) at vm_object_vndeallocate+0xbd > vm_object_deallocate(c17e7210,c17e7210,0,c0703dc4,89e) at > vm_object_deallocate+0x85 > vm_map_entry_delete(c19cbbb8,c16258c4,284c8000,c19cbbb8,c16258c4) at > vm_map_entry_delete+0x130 > vm_map_delete(c19cbbb8,284c8000,284c9000,0,c1996450) at > vm_map_delete+0x18f > munmap(c1996450,cb6ecd14,2,3,292) at munmap+0xa3 > syscall(280b002f,2f,bfbf002f,2831e058,815f2d8) at syscall+0x213 > Xint0x80_syscall() at Xint0x80_syscall+0x1f > --- syscall (73, FreeBSD ELF32, munmap), eip = 0x28393cbb, esp = > 0xbfbfc6fc, ebp = 0xbfbfc718 --- > > Another httpd holding a file lock, blocked in suspfs from ufs_inactive: > > db> trace 3047 > Tracing pid 3047 tid 100143 td 0xc19b2450 > sched_switch(c19b2450,0,1) at sched_switch+0x17f > mi_switch(1,0) at mi_switch+0x264 > sleepq_switch(c137086c,cb781b8c,c0526901,c137086c,0) at > sleepq_switch+0xe0 > sleepq_wait(c137086c,0,0,1,c141ec08) at sleepq_wait+0x30 > msleep(c137086c,c1370844,29f,c06f2e66,0) at msleep+0x2d1 > vn_write_suspend_wait(c18419b4,c1370800,1,0,c06f2634) at > vn_write_suspend_wait+0xe4 > ufs_inactive(cb781c14) at ufs_inactive+0x186 > VOP_INACTIVE_APV(c076ed60,cb781c14) at VOP_INACTIVE_APV+0x7e > vrele(c18419b4,c1826000,0,c07046ce,19d) at vrele+0x10e > vm_object_vndeallocate(c1826000) at vm_object_vndeallocate+0xbd > vm_object_deallocate(c1826000,c1826000,0,c0703dc4,89e) at > vm_object_deallocate+0x85 > vm_map_entry_delete(c19cbe10,c18bfd8c,284cd000,c19cbe10,c18bfd8c) at > vm_map_entry_delete+0x130 > vm_map_delete(c19cbe10,284cd000,284ce000,0,c19b2450) at > vm_map_delete+0x18f > munmap(c19b2450,cb781d14,2,3,292) at munmap+0xa3 > syscall(280b002f,2f,bfbf002f,2831e058,815d228) at syscall+0x213 > Xint0x80_syscall() at Xint0x80_syscall+0x1f > --- syscall (73, FreeBSD ELF32, munmap), eip = 0x28393cbb, esp = > 0xbfbfc6fc, ebp = 0xbfbfc718 --- > > mksnap_ffs, creating the snapshot, but blocked on a vnode lock: > > db> trace 2993 > Tracing pid 2993 tid 100096 td 0xc162d8a0 > sched_switch(c162d8a0,0,1) at sched_switch+0x17f > mi_switch(1,0) at mi_switch+0x264 > sleepq_switch(c13c4a78,cb6b560c,c0526901,c13c4a78,0) at > sleepq_switch+0xe0 > sleepq_wait(c13c4a78,0,0,c13c4a78,1010040) at sleepq_wait+0x30 > msleep(c13c4a78,c0784458,50,c06f08fe,0) at msleep+0x2d1 > acquire(cb6b5664,1010040,600,c162d8a0,0) at acquire+0x9a > lockmgr(c13c4a78,1010002,c13c49b4,c162d8a0,cb6b5684) at lockmgr+0x382 > vop_stdlock(cb6b56b4) at vop_stdlock+0x1b > VOP_LOCK_APV(c076f340,cb6b56b4,cb6b56a4,c06c279e,cb6b56b4) at > VOP_LOCK_APV+0x7e > ffs_lock(cb6b56b4) at ffs_lock+0x3b > VOP_LOCK_APV(c076ed60,cb6b56b4) at VOP_LOCK_APV+0x7e > vn_lock(c13c49b4,10002,c162d8a0) at vn_lock+0xc2 > ffs_snapshot(c1370800,c14ccbc0,c14ccbc0,c1369b00,0) at > ffs_snapshot+0x1242 > ffs_mount(c1370800,c162d8a0,20000000,201000,c13c49b4) at > ffs_mount+0x7a5 > vfs_domount(c162d8a0,c13207b0,c141c030,1211000,c1320670) at > vfs_domount+0x576 > vfs_donmount(c162d8a0,1211000,cb6b5c14,c1422880,e) at vfs_donmount+0xce > kernel_mount(c1320690,1211000,bfbfed33,0,0) at kernel_mount+0x6d > ffs_cmount(c1320690,bfbfe578,1211000,c162d8a0,c076ea80) at > ffs_cmount+0x5d > mount(c162d8a0,cb6b5d14,4,2,206) at mount+0x156 > syscall(2f,2f,2f,bfbfe7c8,bfbfed29) at syscall+0x213 > Xint0x80_syscall() at Xint0x80_syscall+0x1f > --- syscall (21, FreeBSD ELF32, mount), eip = 0x280b80db, esp = > 0xbfbfe4ec, ebp = 0xbfbfebe0 --- > > Anothr httpd blocked in suspfs while holding a file vnode lock, trying > to > unmap: > > db> trace 2609 > Tracing pid 2609 tid 100108 td 0xc1807b80 > sched_switch(c1807b80,0,1) at sched_switch+0x17f > mi_switch(1,0) at mi_switch+0x264 > sleepq_switch(c137086c,cb70ab8c,c0526901,c137086c,0) at > sleepq_switch+0xe0 > sleepq_wait(c137086c,0,0,1,c14cae38) at sleepq_wait+0x30 > msleep(c137086c,c1370844,29f,c06f2e66,0) at msleep+0x2d1 > vn_write_suspend_wait(c19fd9b4,c1370800,1,0,c06f2634) at > vn_write_suspend_wait+0xe4 > ufs_inactive(cb70ac14) at ufs_inactive+0x186 > VOP_INACTIVE_APV(c076ed60,cb70ac14) at VOP_INACTIVE_APV+0x7e > vrele(c19fd9b4,c1a93948,0,c07046ce,19d) at vrele+0x10e > vm_object_vndeallocate(c1a93948) at vm_object_vndeallocate+0xbd > vm_object_deallocate(c1a93948,c1a93948,0,c0703dc4,89e) at > vm_object_deallocate+0x85 > vm_map_entry_delete(c18bd834,c18cb3b8,284c7000,c18bd834,c18cb3b8) at > vm_map_entry_delete+0x130 > vm_map_delete(c18bd834,284c7000,284c8000,0,c1807b80) at > vm_map_delete+0x18f > munmap(c1807b80,cb70ad14,2,37,296) at munmap+0xa3 > syscall(2f,2f,bfbf002f,2831e058,815d0c8) at syscall+0x213 > Xint0x80_syscall() at Xint0x80_syscall+0x1f > --- syscall (73, FreeBSD ELF32, munmap), eip = 0x28393cbb, esp = > 0xbfbfc5cc, ebp = 0xbfbfc5e8 --- > > No WITNESS in this kernel. > > My first glance reading is that there's a deadlock between the snapshot > lock and vnode locks, which ocurs because a vnode lock has been > acquired > as part of the mmap tear-down, which leads to vn_write_suspend() being > called in ufs_inactive as the inode has a zero reference count so has > to > be GC'd, which involves writing to the file system. VOP_INACTIVE is > called by vrele() with the vnode lock held. My understanding is that, > generally, we acquire the file system snapshot lock before vnode locks > as > part of the VFS lock order. > > This could be a misinterpretation. I'll try to get a dump, but this > is a > twe box so I may not be able to. > > Robert N M Watson > > > _______________________________________________ > freebsd-current_at_freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-current > To unsubscribe, send any mail to > "freebsd-current-unsubscribe_at_freebsd.org"Received on Sat Feb 19 2005 - 16:51:31 UTC
This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:38:28 UTC