Re: VFS deadlock using dump -L on HEAD -- VOP_INACTIVE acquires snapshot file system suspend lock after vnode lock on delete-after-last-close?

From: Frode Nordahl <frode_at_nordahl.net>
Date: Sat, 19 Feb 2005 18:51:22 +0100
On Feb 18, 2005, at 15:49, Robert Watson wrote:

>
> I upgraded a box to a recent 6.x this morning, and ran into the 
> following
> apparently deadlock while backing up a partitionusing dump -L.  It 
> appears
> to have gotten stuck during mksnap_ffs.  Some interpretation below, but
> the high level summary is that it looks to me like we have a lock order
> issue between the snapshot write lock for the file system and vnode 
> locks,
> which occurs because vrele() calls VOP_INACTIVE with the vnode lock 
> held,
> and ufs_inactive() then proceeds to get a write reference to the file
> system.  It could be a fix is to grab the write reference to the file
> system in vrele() before the vnode lock, to aovid grabbing the write
> reference in ufs_inactive().

My 5.2.1 and RELENG_5 NFS servers all hang up every time I try to use 
dump -L too.

However, background fsck, with some exceptions allways works. One time 
a system came up after a crash just before the nightly periodic scripts 
where to be run (lots of finds running etc), and that made it deadlock 
in a similar way. But except for this incident I have made good use of 
background fsck countless times.

I have one of the NFS servers on serial console, so I decided to retry 
a dump -L now with RELENG_5, since I am doing some changes to our 
backup routines anyway.

I completely forgot do call doadump though, so I probably lost some 
valuable information :-(

Here is what I extracted:
db> show lockedvnods
Locked vnodes
0xc418ca50: tag ufs, type VDIR, usecount 3, writecount 0, refcount 1, 
flags (VV_OBJBUF), lock type ufs: EXCL (count 1) by thread 0xc8e1c320 
(pid 89338)
         ino 3, on dev da1s1c (4, 18)
0xc66f1528: tag ufs, type VREG, usecount 2, writecount 0, refcount 
4103, lock type ufs: EXCL (count 1) by thread 0xc3e4a7d0 (pid 88381) 
with 1 pending
         ino 274, on dev da1s1c (4, 18)
0xc799a630: tag ufs, type VREG, usecount 1, writecount 0, refcount 147, 
flags (VV_OBJBUF), lock type ufs: EXCL (count 1) by thread 0xc3914af0 
(pid 418) with 2 pending
         ino 41430101, on dev da1s1c (4, 18)
db> ps
   pid   proc     uarea   uid  ppid  pgrp  flag   stat  wmesg    wchan  
cmd
89351 c3def710 ee183000    0  5842 89351 0004002 [SLPQ ttyin 
0xc90a9c10][SLP] bash
89338 c8e1954c ee2e2000    0 89255 89338 0004002 [SLPQ ufs 
0xc66f15d4][SLP] ls
89255 c3defe20 ee187000    0  5842 89255 0004002 [SLPQ wait 
0xc3defe20][SLP] bash
88630 c3dea388 ee11c000    0 85239 88630 0004002 [SLPQ select 
0xc08efcc4][SLP] top
88381 c3e47e20 ee229000    0 88380 88298 0004002 [SLPQ ufs 
0xc799a6dc][SLP] mksnap_ffs
88380 c3e4c000 ee233000    0 88379 88298 0004002 [SLPQ wait 
0xc3e4c000][SLP] sh
88379 c398b388 ee061000    0 88298 88298 0004002 [SLPQ wait 
0xc398b388][SLP] dump
88298 c3bd5000 ee078000    0  5843 88298 0004002 [SLPQ wait 
0xc3bd5000][SLP] sh
85239 c3e4c54c ee236000    0  5842 85239 0004002 [SLPQ wait 
0xc3e4c54c][SLP] bash
83213 c3e47a98 ee227000    0 83201 83213 0004002 [SLPQ ttyin 
0xc97a0c10][SLP] bash
83201 c3e4c388 ee235000    0   496 83201 0000100 [SLPQ select 
0xc08efcc4][SLP] sshd
82285 c3e47c5c ee228000    0 81759 82285 0004002 [SLPQ pause 
0xc3e47c94][SLP] screen
81759 c3df88d4 ee1df000    0 81752 81759 0004002 [SLPQ wait 
0xc3df88d4][SLP] bash
81752 c3df8388 ee1dc000    0   496 81752 0000100 [SLPQ select 
0xc08efcc4][SLP] sshd
  5843 c3def8d4 ee184000    0  5842  5843 0004002 [SLPQ wait 
0xc3def8d4][SLP] bash
  5842 c3e47388 ee1e5000    0     1  5842 0000000 [SLPQ select 
0xc08efcc4][SLP] screen
13643 c3df8a98 ee1e0000    0     1 13643 0004002 [SLPQ ttyin 
0xc38fe210][SLP] getty
   920 c3deaa98 ee17c000    0   564   920 0004002 [SLPQ ttyin 
0xc3619e10][SLP] bash
   569 c3bda54c ee0de000    0     1   569 0004002 [SLPQ ttyin 
0xc3680c10][SLP] getty
   568 c3915c5c ebe2a000    0     1   568 0004002 [SLPQ ttyin 
0xc3680e10][SLP] getty
   567 c3bdaa98 ee0e1000    0     1   567 0004002 [SLPQ ttyin 
0xc3681010][SLP] getty
   566 c3bd58d4 ee0d7000    0     1   566 0004002 [SLPQ ttyin 
0xc3681210][SLP] getty
   565 c398b000 ee05f000    0     1   565 0004002 [SLPQ ttyin 
0xc3634a10][SLP] getty
   564 c398be20 ee067000    0     1   564 0004102 [SLPQ wait 
0xc398be20][SLP] login
   563 c3986e20 ee05e000    0     1   563 0004002 [SLPQ ttyin 
0xc3619810][SLP] login
   551 c39121c4 ebdfd000    0     1   551 0000000 [SLPQ select 
0xc08efcc4][SLP] inetd
   526 c3bd5388 ee098000    0     1   114 0000002 [SLPQ nanslp 
0xc08ea56c][SLP] perl
   518 c3bd5a98 ee0d8000    0     1   518 0000000 [SLPQ nanslp 
0xc08ea56c][SLP] cron
   506 c3bda388 ee0dd000   25     1   506 0000100 [SLPQ pause 
0xc3bda3c0][SLP] sendmail
   502 c3bd51c4 ee07f000    0     1   502 0000100 [SLPQ select 
0xc08efcc4][SLP] sendmail
   496 c398b1c4 ee060000    0     1   496 0000100 [SLPQ select 
0xc08efcc4][SLP] sshd
   485 c3bd5c5c ee0d9000    1     1   485 0000100 [SLPQ sbwait 
0xc3bb3974][SLP] rwhod
   474 c39861c4 ee057000    0     1   474 0000000 [SLPQ select 
0xc08efcc4][SLP] ntpd
   450 c398654c ee059000    0     1   450 0000000 [SLPQ select 
0xc08efcc4][SLP] usbd
   435 c3912710 ebe00000    0   429   429 0000100 [SLPQ select 
0xc08efcc4][SLP] rpc.lockd
   429 c398b54c ee062000    0     1   429 0000000 [SLPQ select 
0xc08efcc4][SLP] rpc.lockd
   424 c3912000 ebdfc000    0     1   424 0000000 [SLPQ select 
0xc08efcc4][SLP] rpc.statd
   418 c3912a98 ebe20000    0   415   415 0000000 [SLPQ suspfs 
0xc38ef46c][SLP] nfsd
   417 c398b8d4 ee064000    0   415   415 0000000 [SLPQ suspfs 
0xc38ef46c][SLP] nfsd
   416 c391254c ebdff000    0   415   415 0000000 [SLPQ suspfs 
0xc38ef46c][SLP] nfsd
   415 c39868d4 ee05b000    0     1   415 0000000 [SLPQ ufs 
0xc799a6dc][SLP] nfsd
   413 c39128d4 ebe01000    0     1   413 0000000 [SLPQ select 
0xc08efcc4][SLP] mountd
   344 c3915e20 ebe2b000    0     1   344 0000000 [SLPQ select 
0xc08efcc4][SLP] rpcbind
   325 c35f1e20 ebd27000    0     1   325 0000000 [SLPQ select 
0xc08efcc4][SLP] syslogd
   305 c3986388 ee058000    0     1   305 0000000 [SLPQ select 
0xc08efcc4][SLP] devd
   113 c3915000 ebe23000    0     0     0 0000204 [SLPQ - 
0xebe10d18][SLP] schedcpu
   112 c39151c4 ebe24000    0     0     0 0000204 [SLPQ - 
0xc08f76ec][SLP] nfsiod 3
   111 c3915388 ebe25000    0     0     0 0000204 [SLPQ - 
0xc08f76e8][SLP] nfsiod 2
   110 c391554c ebe26000    0     0     0 0000204 [SLPQ - 
0xc08f76e4][SLP] nfsiod 1
   109 c3915710 ebe27000    0     0     0 0000204 [SLPQ - 
0xc08f76e0][SLP] nfsiod 0
   108 c39158d4 ebe28000    0     0     0 0000204 [SLPQ hptrdy 
0xc078fda4][SLP] hpt_wt
   107 c353c1c4 ebcdb000    0     0     0 0000204 [SLPQ vlruwt 
0xc353c1c4][SLP] vnlru
   106 c353c388 ebcdc000    0     0     0 0000204 [SLPQ syncer 
0xc08ea2ec][SLP] syncer
   105 c353c54c ebcdd000    0     0     0 0000204 [SLPQ psleep 
0xc08f028c][SLP] bufdaemon
   104 c353c710 ebcde000    0     0     0 000020c [SLPQ pgzero 
0xc08fe054][SLP] pagezero
   103 c353c8d4 ebcdf000    0     0     0 0000204 [SLPQ psleep 
0xc08fe0a8][SLP] vmdaemon
   102 c353ca98 ebce0000    0     0     0 0000204 [SLPQ psleep 
0xc08fe064][SLP] pagedaemon
   101 c353cc5c ebce1000    0     0     0 0000204 [IWAIT] swi0: sio
   100 c353ce20 ebd1e000    0     0     0 0000204 [SLPQ - 
0xc35f303c][SLP] fdc0
    99 c35f1000 ebd1f000    0     0     0 0000204 [SLPQ usbevt 
0xc362b210][SLP] usb2
    98 c35f11c4 ebd20000    0     0     0 0000204 [SLPQ usbevt 
0xc3627210][SLP] usb1
    97 c35f1388 ebd21000    0     0     0 0000204 [SLPQ usbtsk 
0xc08e2138][SLP] usbtask
    96 c35f154c ebd22000    0     0     0 0000204 [SLPQ usbevt 
0xc3611210][SLP] usb0
    95 c35f1710 ebd23000    0     0     0 0000204 [SLPQ idle 
0xc361b000][SLP] aic_recovery1
    94 c35f18d4 ebd24000    0     0     0 0000204 [SLPQ idle 
0xc3617000][SLP] aic_recovery0
     9 c35f1a98 ebd25000    0     0     0 0000204 [SLPQ actask 
0xc0a34a2c][SLP] acpi_task2
     8 c35f1c5c ebd26000    0     0     0 0000204 [SLPQ actask 
0xc0a34a2c][SLP] acpi_task1
     7 c351f54c ebcad000    0     0     0 0000204 [SLPQ actask 
0xc0a34a2c][SLP] acpi_task0
    93 c351f710 ebcae000    0     0     0 0000204 [IWAIT] swi6:+
     6 c351f8d4 ebcaf000    0     0     0 0000204 [SLPQ - 
0xc358fbc0][SLP] thread taskq
    92 c351fa98 ebcb0000    0     0     0 0000204 [IWAIT] swi6: acpitaskq
    91 c351fc5c ebcb1000    0     0     0 0000204 [IWAIT] swi6:+
    90 c351fe20 ebcb2000    0     0     0 0000204 [IWAIT] swi6: task 
queue
     5 c353a000 ebcb3000    0     0     0 0000204 [SLPQ - 
0xc358fd80][SLP] kqueue taskq
    89 c353a1c4 ebcd2000    0     0     0 0000204 [IWAIT] swi3: cambio
    88 c353a388 ebcd3000    0     0     0 0000204 [IWAIT] swi2: camnet
    87 c353a54c ebcd4000    0     0     0 0000204 [SLPQ - 
0xc08dfe80][SLP] yarrow
     4 c353a710 ebcd5000    0     0     0 0000204 [SLPQ - 
0xc08e4b28][SLP] g_down
     3 c353a8d4 ebcd6000    0     0     0 0000204 [SLPQ - 
0xc08e4b24][SLP] g_up
     2 c353aa98 ebcd7000    0     0     0 0000204 [SLPQ - 
0xc08e4b1c][SLP] g_event
    86 c353ac5c ebcd8000    0     0     0 0000204 [IWAIT] swi4: vm
    85 c353ae20 ebcd9000    0     0     0 000020c [IWAIT] swi5: clock sio
    84 c353c000 ebcda000    0     0     0 0000204 [IWAIT] swi1: net
    83 c350da98 ebc62000    0     0     0 0000204 [IWAIT] irq0: clk
    82 c350dc5c ebc63000    0     0     0 0000204 [IWAIT] irq71:
    81 c350de20 ebc64000    0     0     0 0000204 [IWAIT] irq70:
    80 c351b000 ebc65000    0     0     0 0000204 [IWAIT] irq69:
    79 c351b1c4 ebc66000    0     0     0 0000204 [IWAIT] irq68:
    78 c351b388 ebc67000    0     0     0 0000204 [IWAIT] irq67:
    77 c351b54c ebca4000    0     0     0 0000204 [IWAIT] irq66:
    76 c351b710 ebca5000    0     0     0 0000204 [IWAIT] irq65:
    75 c351b8d4 ebca6000    0     0     0 0000204 [IWAIT] irq64:
    74 c351ba98 ebca7000    0     0     0 0000204 [IWAIT] irq63:
    73 c351bc5c ebca8000    0     0     0 0000204 [IWAIT] irq62:
    72 c351be20 ebca9000    0     0     0 0000204 [IWAIT] irq61:
    71 c351f000 ebcaa000    0     0     0 0000204 [IWAIT] irq60:
    70 c351f1c4 ebcab000    0     0     0 0000204 [IWAIT] irq59: em1
    69 c351f388 ebcac000    0     0     0 0000204 [IWAIT] irq58: em0
    68 c34fc1c4 ebc18000    0     0     0 0000204 [IWAIT] irq57:
    67 c34fc388 ebc19000    0     0     0 0000204 [IWAIT] irq56:
    66 c34fc54c ebc1a000    0     0     0 0000204 [IWAIT] irq55:
    65 c34fc710 ebc1b000    0     0     0 0000204 [IWAIT] irq54:
    64 c34fc8d4 ebc1c000    0     0     0 0000204 [IWAIT] irq53:
    63 c34fca98 ebc59000    0     0     0 0000204 [IWAIT] irq52: iir0
    62 c34fcc5c ebc5a000    0     0     0 0000204 [IWAIT] irq51:
    61 c34fce20 ebc5b000    0     0     0 0000204 [IWAIT] irq50:
    60 c350d000 ebc5c000    0     0     0 0000204 [IWAIT] irq49:
    59 c350d1c4 ebc5d000    0     0     0 0000204 [IWAIT] irq48:
    58 c350d388 ebc5e000    0     0     0 0000204 [IWAIT] irq47:
    57 c350d54c ebc5f000    0     0     0 0000204 [IWAIT] irq46:
    56 c350d710 ebc60000    0     0     0 0000204 [IWAIT] irq45:
    55 c350d8d4 ebc61000    0     0     0 0000204 [IWAIT] irq44:
    54 c34e6a98 ebbed000    0     0     0 0000204 [IWAIT] irq43:
    53 c34e6c5c ebbee000    0     0     0 0000204 [IWAIT] irq42:
    52 c34e6e20 ebbef000    0     0     0 0000204 [IWAIT] irq41:
    51 c34f8000 ebbf0000    0     0     0 0000204 [IWAIT] irq40:
    50 c34f81c4 ebc0f000    0     0     0 0000204 [IWAIT] irq39:
    49 c34f8388 ebc10000    0     0     0 0000204 [IWAIT] irq38:
    48 c34f854c ebc11000    0     0     0 0000204 [IWAIT] irq37:
    47 c34f8710 ebc12000    0     0     0 0000204 [IWAIT] irq36:
    46 c34f88d4 ebc13000    0     0     0 0000204 [IWAIT] irq35:
    45 c34f8a98 ebc14000    0     0     0 0000204 [IWAIT] irq34:
    44 c34f8c5c ebc15000    0     0     0 0000204 [IWAIT] irq33:
    43 c34f8e20 ebc16000    0     0     0 0000204 [IWAIT] irq32:
    42 c34fc000 ebc17000    0     0     0 0000204 [IWAIT] irq31: ahd1
    41 c34dc54c ebbc3000    0     0     0 0000204 [IWAIT] irq30: ahd0
    40 c34dc710 ebbc4000    0     0     0 0000204 [IWAIT] irq29:
    39 c34dc8d4 ebbc5000    0     0     0 0000204 [IWAIT] irq28:
    38 c34dca98 ebbe4000    0     0     0 0000204 [IWAIT] irq27:
    37 c34dcc5c ebbe5000    0     0     0 0000204 [IWAIT] irq26:
    36 c34dce20 ebbe6000    0     0     0 0000204 [IWAIT] irq25:
    35 c34e6000 ebbe7000    0     0     0 0000204 [IWAIT] irq24:
    34 c34e61c4 ebbe8000    0     0     0 0000204 [IWAIT] irq23:
    33 c34e6388 ebbe9000    0     0     0 0000204 [IWAIT] irq22:
    32 c34e654c ebbea000    0     0     0 0000204 [IWAIT] irq21:
    31 c34e6710 ebbeb000    0     0     0 0000204 [IWAIT] irq20:
    30 c34e68d4 ebbec000    0     0     0 0000204 [IWAIT] irq19: uhci1
    29 c348c1c4 ea178000    0     0     0 0000204 [IWAIT] irq18: uhci2
    28 c348c388 ea179000    0     0     0 0000204 [IWAIT] irq17:
    27 c348c54c ea198000    0     0     0 0000204 [IWAIT] irq16: uhci0
    26 c348c710 ea199000    0     0     0 0000204 [IWAIT] irq15: ata1
    25 c348c8d4 ea19a000    0     0     0 0000204 [IWAIT] irq14: ata0
    24 c348ca98 ea19b000    0     0     0 0000204 [IWAIT] irq13:
    23 c348cc5c ea19c000    0     0     0 0000204 [IWAIT] irq12:
    22 c348ce20 ea19d000    0     0     0 0000204 [IWAIT] irq11:
    21 c34dc000 ebbc0000    0     0     0 0000204 [IWAIT] irq10:
    20 c34dc1c4 ebbc1000    0     0     0 0000204 [IWAIT] irq9: acpi0
    19 c34dc388 ebbc2000    0     0     0 0000204 [IWAIT] irq8: rtc
    18 c3482000 ea132000    0     0     0 0000204 [IWAIT] irq7: ppc0
    17 c34821c4 ea16f000    0     0     0 0000204 [IWAIT] irq6: fdc0
    16 c3482388 ea170000    0     0     0 0000204 [IWAIT] irq5:
    15 c348254c ea171000    0     0     0 0000204 [IWAIT] irq4: sio0
    14 c3482710 ea172000    0     0     0 0000204 [IWAIT] irq3: sio1
    13 c34828d4 ea173000    0     0     0 0000204 [IWAIT] irq1: atkbd0
    12 c3482a98 ea174000    0     0     0 000020c [CPU 0] idle: cpu0
    11 c3482c5c ea175000    0     0     0 000020c [CPU 1] idle: cpu1
     1 c3482e20 ea176000    0     0     1 0004200 [SLPQ wait 
0xc3482e20][SLP] init
    10 c348c000 ea177000    0     0     0 0000204 [SLPQ ktrace 
0xc08e8518][SLP] ktrace
     0 c08e4c20 c0c1f000    0     0     0 0000200 [SLPQ sched 
0xc08e4c20][SLP] swapper
db> trace 88381
Tracing pid 88381 tid 100235 td 0xc3e4a7d0
sched_switch(c3e4a7d0,0,1) at sched_switch+0x143
mi_switch(1,0,c3e4a7d0,1,c3e4a7d0) at mi_switch+0x1ba
sleepq_switch(c799a6dc) at sleepq_switch+0x133
sleepq_wait(c799a6dc,0,f3,0,c3c774a0) at sleepq_wait+0xb
msleep(c799a6dc,c08e93c4,50,c08303e7,0) at msleep+0x322
acquire(ee21a7c4,1000040,600,c3e4a7d0,0) at acquire+0xa2
lockmgr(c799a6dc,1010002,c799a630,c3e4a7d0,10002) at lockmgr+0x366
ufs_lock(ee21a7f0,ee21a80c,c0674b20,ee21a7f0,c08ca8e0) at ufs_lock+0x3c
ufs_vnoperate(ee21a7f0) at ufs_vnoperate+0x13
vn_lock(c799a630,10002,c3e4a7d0) at vn_lock+0x104
ffs_snapshot(c38ef400,bfbfe9eb,c3911a00,ee21aab0,0) at 
ffs_snapshot+0x11d2
ffs_omount(c38ef400,c9a96480,bfbfe248,c3e4a7d0,0) at ffs_omount+0x52e
vfs_domount(c3e4a7d0,c8cc0990,c9a96480,1211118,bfbfe248) at 
vfs_domount+0x7b8
mount(c3e4a7d0,ee21ad14,4,0,206) at mount+0x6c
syscall(2f,2f,2f,bfbfe498,bfbfe9e1) at syscall+0x283
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (21, FreeBSD ELF32, mount), eip = 0x280c1473, esp = 
0xbfbfe1bc, ebp = 0xbfbfe8b0 ---
db> trace 415
Tracing pid 415 tid 100143 td 0xc39884b0
sched_switch(c39884b0,0,1) at sched_switch+0x143
mi_switch(1,0,c39884b0,1,c39884b0) at mi_switch+0x1ba
sleepq_switch(c799a6dc) at sleepq_switch+0x133
sleepq_wait(c799a6dc,0,0,0,0) at sleepq_wait+0xb
msleep(c799a6dc,c08e93c4,50,c08303e7,0) at msleep+0x322
acquire(ee0448ec,1000040,600,c39884b0,0) at acquire+0xa2
lockmgr(c799a6dc,1010002,c799a630,c39884b0,10002) at lockmgr+0x366
ufs_lock(ee044918,ee044934,c0674b20,ee044918,c08ca8e0) at ufs_lock+0x3c
ufs_vnoperate(ee044918) at ufs_vnoperate+0x13
vn_lock(c799a630,10002,c39884b0,2782c55,4) at vn_lock+0x104
vget(c799a630,10002,c39884b0,c799a630,c39884b0) at vget+0xfd
ufs_ihashget(c393dd00,2782c55,2,ee044a0c,1010002) at ufs_ihashget+0xfe
ffs_vget(c38ef400,2782c55,2,ee044a0c) at ffs_vget+0x31
ufs_fhtovp(c38ef400,ee044bf8,ee044ae4,c38ef400,ee044a70) at 
ufs_fhtovp+0x22
ffs_fhtovp(c38ef400,ee044bf8,ee044ae4,c39884b0,100) at ffs_fhtovp+0x3a
nfsrv_fhtovp(ee044bf0,1,ee044ae4,ca071380,c3bb1780) at nfsrv_fhtovp+0xf5
nfsrv_read(ca071300,c3bb1780,c39884b0,ee044ca8,ee044ca8) at 
nfsrv_read+0x25d
nfssvc_nfsd(c39884b0,0,c39884b0,ee044ce8,c39868d4) at nfssvc_nfsd+0x406
nfssvc(c39884b0,ee044d14,2,1,296) at nfssvc+0x1bc
syscall(2f,2f,2f,bfbfedf4,8052180) at syscall+0x283
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (155, FreeBSD ELF32, nfssvc), eip = 0x280c2ab3, esp = 
0xbfbfeb1c, ebp = 0xbfbfeb38 ---
db> trace 416
Tracing pid 416 tid 100098 td 0xc353bc80
sched_switch(c353bc80,0,1) at sched_switch+0x143
mi_switch(1,0,c353bc80,1,c353bc80) at mi_switch+0x1ba
sleepq_switch(c38ef46c) at sleepq_switch+0x133
sleepq_wait(c38ef46c,0,801031d,0,0) at sleepq_wait+0xb
msleep(c38ef46c,0,9f,c0831fcf,0) at msleep+0x322
vn_start_write(0,ebcceabc,1) at vn_start_write+0x8b
nfsrv_write(ca071900,c3bb1780,c353bc80,ebcceca8,ebcceca0) at 
nfsrv_write+0x1b4
nfssvc_nfsd(c353bc80,0,c353bc80,ebccece8,c391254c) at nfssvc_nfsd+0x406
nfssvc(c353bc80,ebcced14,2,0,296) at nfssvc+0x1bc
syscall(2f,2f,2f,bfbfeec4,3) at syscall+0x283
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (155, FreeBSD ELF32, nfssvc), eip = 0x280c2ab3, esp = 
0xbfbfeb1c, ebp = 0xbfbfeb38 ---
db> trace 417
Tracing pid 417 tid 100134 td 0xc3987640
sched_switch(c3987640,0,1) at sched_switch+0x143
mi_switch(1,0,c3987640,1,c3987640) at mi_switch+0x1ba
sleepq_switch(c38ef46c) at sleepq_switch+0x133
sleepq_wait(c38ef46c,0,3a,0,c364bba0) at sleepq_wait+0xb
msleep(c38ef46c,0,9f,c0831fcf,0) at msleep+0x322
vn_start_write(0,ee029abc,1) at vn_start_write+0x8b
nfsrv_write(ca2e2000,c3bb1780,c3987640,ee029ca8,ee029ca0) at 
nfsrv_write+0x1b4
nfssvc_nfsd(c3987640,0,c3987640,ee029ce8,c398b8d4) at nfssvc_nfsd+0x406
nfssvc(c3987640,ee029d14,2,0,296) at nfssvc+0x1bc
syscall(2f,2f,2f,bfbfeec4,3) at syscall+0x283
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (155, FreeBSD ELF32, nfssvc), eip = 0x280c2ab3, esp = 
0xbfbfeb1c, ebp = 0xbfbfeb38 ---
db> trace 418
Tracing pid 418 tid 100127 td 0xc3914af0
sched_switch(c3914af0,0,1) at sched_switch+0x143
mi_switch(1,0,c3914af0,1,c3914af0) at mi_switch+0x1ba
sleepq_switch(c38ef46c) at sleepq_switch+0x133
sleepq_wait(c38ef46c,0,c08ca3e0,0,c364bba0) at sleepq_wait+0xb
msleep(c38ef46c,0,9f,c0831fcf,0) at msleep+0x322
vn_write_suspend_wait(c799a630,c38ef400,1,0,dc59b08c) at 
vn_write_suspend_wait+0x89
ufs_inactive(ebe19a74,ebe19a8c,c0669ae7,ebe19a74,c08ca860) at 
ufs_inactive+0x1ba
ufs_vnoperate(ebe19a74) at ufs_vnoperate+0x13
vput(c799a630) at vput+0xfb
nfsrv_read(ca10f000,c3bb1780,c3914af0,ebe19ca8,ebe19ca0) at 
nfsrv_read+0xcfa
nfssvc_nfsd(c3914af0,0,c3914af0,ebe19ce8,c3912a98) at nfssvc_nfsd+0x406
nfssvc(c3914af0,ebe19d14,2,0,296) at nfssvc+0x1bc
syscall(2f,2f,2f,bfbfeec4,3) at syscall+0x283
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (155, FreeBSD ELF32, nfssvc), eip = 0x280c2ab3, esp = 
0xbfbfeb1c, ebp = 0xbfbfeb38 ---
db>


Regards,
Frode Nordahl

> Robert N M Watson
>
> www:~> more backup.csh
> #!/bin/csh
>
> set DATE=`date +"%Y%m%d"`
>
> dump -L -f - -a -u -0 /home/pnn | gzip -9 > ${DATE}-backup.dump.gz
> www:~> df
> Filesystem    1K-blocks    Used    Avail Capacity  Mounted on
> /dev/twed0s1a    507630  103502   363518    22%    /
> devfs                 1       1        0   100%    /dev
> /dev/twed0s1f  89359640  370128 81840742     0%    /home/pnn
> /dev/twed0s1e  16244334 3211132 11733656    21%    /usr
> /dev/twed0s1d   8122126 1541978  5930378    21%    /var
> www:~> ./backup.csh
> load: 0.00  cmd: mksnap_ffs 2993 [ufs] 0.00u 3.88s 0% 496k
> load: 0.00  cmd: mksnap_ffs 2993 [ufs] 0.00u 3.88s 0% 496k
> load: 0.00  cmd: mksnap_ffs 2993 [ufs] 0.00u 3.88s 0% 496k
> load: 0.00  cmd: mksnap_ffs 2993 [ufs] 0.00u 3.88s 0% 496k
>
> On the serial console:
>
> telnet> send break
> KDB: enter: Line break on console
> [thread pid 11 tid 100004 ]
> Stopped at      kdb_enter+0x2b: nop
> db> show lockedvnods
> Locked vnodes
>
> 0xc13c49b4: tag ufs, type VDIR
>     usecount 153, writecount 0, refcount 1 mountedhere 0
>     flags (VV_ROOT)
>     v_object 0
>      lock type ufs: EXCL (count 1) by thread 0xc1807a10 (pid 3057) with
> 152 pend
> ing
>         ino 2, on dev twed0s1f (229, 9)
>
> 0xc15268a0: tag ufs, type VDIR
>     usecount 2, writecount 0, refcount 1 mountedhere 0
>     flags ()
>     v_object 0
>      lock type ufs: EXCL (count 1) by thread 0xc137e000 (pid 3062) 
> with 1
> pendin
> g
>         ino 3085322, on dev twed0s1f (229, 9)
>
> 0xc163bbdc: tag ufs, type VDIR
>     usecount 2, writecount 0, refcount 1 mountedhere 0
>     flags ()
>     v_object 0
>      lock type ufs: EXCL (count 1) by thread 0xc162dcf0 (pid 3056) 
> with 1
> pendin
> g
>         ino 3133686, on dev twed0s1f (229, 9)
>
> 0xc17e933c: tag ufs, type VREG
>     usecount 1, writecount 0, refcount 0 mountedhere 0
>     flags ()
>     v_object 0xc17e7210
>      lock type ufs: EXCL (count 1) by thread 0xc1996450 (pid 3045) 
> with 1
> pendin
> g
>         ino 3134802, on dev twed0s1f (229, 9)
>
> 0xc18419b4: tag ufs, type VREG
>     usecount 0, writecount 0, refcount 0 mountedhere 0
>     flags ()
>     v_object 0xc1826000
>      lock type ufs: EXCL (count 1) by thread 0xc19b2450 (pid 3047)
>         ino 3132762, on dev twed0s1f (229, 9)
>
> 0xc1ac933c: tag ufs, type VREG
>     usecount 1, writecount 0, refcount 865 mountedhere 0
>     flags ()
>     v_object 0
>      lock type ufs: EXCL (count 1) by thread 0xc162d8a0 (pid 2993)
>         ino 9891841, on dev twed0s1f (229, 9)
>
> 0xc19fd9b4: tag ufs, type VREG
>     usecount 0, writecount 0, refcount 0 mountedhere 0
>     flags ()
>     v_object 0xc1a93948
>      lock type ufs: EXCL (count 1) by thread 0xc1807b80 (pid 2609)
>         ino 3197600, on dev twed0s1f (229, 9)
> db> ps
>   pid   proc     uid  ppid  pgrp  flag   stat  wmesg    wchan  cmd
>  3296 c1cf93f8 1062  3292  3292 4000000 [SLPQ ufs 0xc13c4a78][SLP] sh
>  3295 c1cf91fc 1062  3293  3293 4000000 [SLPQ ufs 0xc13c4a78][SLP] sh
>  3293 c1cf9be8 1062  3290  3293 0004000 [SLPQ wait 0xc1cf9be8][SLP] sh
>  3292 c1d66000 1062  3289  3292 0004000 [SLPQ wait 0xc1d66000][SLP] sh
>  3290 c19b0000    0   390   390 0000000 [SLPQ piperd 0xc1655480][SLP] 
> cron
>  3289 c1d661fc    0   390   390 0000000 [SLPQ piperd 0xc19a3d80][SLP] 
> cron
>  3271 c1d66be8 1062  3268  3268 4000000 [SLPQ ufs 0xc13c4a78][SLP] sh
>  3268 c1d69000 1062  3267  3268 0004000 [SLPQ wait 0xc1d69000][SLP] sh
>  3267 c1d669ec    0   390   390 0000000 [SLPQ piperd 0xc13aac00][SLP] 
> cron
>  3247 c1cf95f4 1062  3245  3245 4000000 [SLPQ ufs 0xc13c4a78][SLP] sh
>  3245 c1cf99ec 1062  3243  3245 0004000 [SLPQ wait 0xc1cf99ec][SLP] sh
>  3243 c1cf9de4    0   390   390 0000000 [SLPQ piperd 0xc13aa480][SLP] 
> cron
>  3227 c1d66de4 1062  3222  3222 4000000 [SLPQ ufs 0xc13c4a78][SLP] sh
>  3226 c1c8f1fc 1062  3223  3223 4000000 [SLPQ ufs 0xc13c4a78][SLP] sh
>  3223 c18063f8 1062  3218  3223 0004000 [SLPQ wait 0xc18063f8][SLP] sh
>  3222 c1805000 1062  3217  3222 0004000 [SLPQ wait 0xc1805000][SLP] sh
>  3218 c19ae000    0   390   390 0000000 [SLPQ piperd 0xc13a9a80][SLP] 
> cron
>  3217 c18067f0    0   390   390 0000000 [SLPQ piperd 0xc13a9300][SLP] 
> cron
>  3197 c1c8f3f8   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3196 c1c8f5f4   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3195 c1c8f7f0   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3194 c1c8f9ec   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3193 c1c8fbe8   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3192 c1c8fde4   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3191 c1cf6000   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3190 c1cf61fc   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3189 c1cf63f8   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3188 c1cf65f4   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3187 c1cf67f0   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3186 c1cf69ec   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3185 c1cf6be8   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3184 c1cf6de4   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3183 c1cf9000   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3182 c1c8e1fc   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3181 c1c8e5f4   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3180 c1c8e9ec   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3179 c1c1e3f8   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3178 c1c1e5f4   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3177 c1c1e7f0   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3176 c1c1e9ec   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3175 c1c1ebe8   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3174 c1c1ede4   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3173 c1c88000   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3172 c1c881fc   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3171 c1c883f8   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3170 c1c885f4   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3169 c1c887f0   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3168 c1c889ec   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3167 c1c88be8   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3166 c1c88de4   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3165 c1c8e000 1062  3163  3163 4000000 [SLPQ ufs 0xc13c4a78][SLP] sh
>  3163 c1c8e3f8 1062  3161  3163 0004000 [SLPQ wait 0xc1c8e3f8][SLP] sh
>  3161 c1c8e7f0    0   390   390 0000000 [SLPQ piperd 0xc19a3000][SLP] 
> cron
>  3159 c1c8ebe8   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3158 c1c8ede4   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3157 c1c8f000   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3156 c1bc37f0   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3155 c1bc39ec   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3154 c1bc3be8   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3153 c1bc3de4   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3152 c1c19000   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3151 c1c191fc   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3150 c1c193f8   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3149 c1c195f4   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3148 c1c197f0   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3147 c1c199ec   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3146 c1c19be8   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3145 c1c19de4   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3144 c1c1a000   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3143 c1c1a1fc   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3142 c1c1a3f8   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3141 c1c1a5f4   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3140 c1c1a7f0   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3139 c1c1a9ec   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3138 c1c1abe8   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3137 c1c1ade4   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3136 c1c1e000   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3135 c1c1e1fc   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3134 c1bc1de4   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3133 c1bc1be8   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3132 c1b837f0   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3131 c1b839ec   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3130 c1b83be8   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3129 c1b83de4   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3128 c1bc1000   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3127 c1bc11fc   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3126 c1bc13f8   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3125 c1bc15f4   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3124 c1bc17f0   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3123 c1bc19ec   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3120 c1bc3000   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3119 c1bc31fc   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3118 c1bc33f8   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3117 c1bc35f4   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3116 c1abb7f0   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3115 c1abb9ec   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3114 c1abbbe8   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3113 c1abbde4   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3112 c1b82000   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3111 c1b821fc   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3110 c1b823f8   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3109 c1b825f4   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3108 c1b827f0   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3107 c1b829ec   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3106 c1b82be8   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3105 c1b82de4   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3104 c1b83000   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3103 c1b831fc   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3102 c1b833f8   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3101 c1b835f4   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3100 c19b07f0   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3099 c19b09ec   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3098 c19b0be8   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3097 c19b0de4   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3096 c1aba000   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3095 c1aba1fc   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3094 c1aba3f8   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3093 c1aba5f4   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3092 c1aba7f0   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3091 c1aba9ec   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3090 c1ababe8   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3089 c1abade4   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3088 c1abb000   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3087 c1abb1fc   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3086 c1abb3f8   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3085 c16291fc   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3084 c1abb5f4   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3083 c1510be8   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3082 c162c3f8   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3081 c18061fc   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3080 c19957f0   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3079 c18055f4   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3078 c19aede4   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3077 c19959ec   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3076 c19aebe8   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3075 c16297f0   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3074 c16293f8   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3073 c19ae5f4   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3072 c19ae1fc   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3071 c19951fc   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3070 c19ae7f0   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3069 c12ab7f0   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3068 c1510de4   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3067 c1805be8   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3066 c1994000   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3065 c19ae9ec   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3064 c137d3f8   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3063 c150f000   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3062 c1510000   80  2910   403 0004000 [SLPQ ufs 0xc163bca0][SLP]
> download.cgi
>  3060 c19953f8   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3059 c19947f0   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3058 c16299ec   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3057 c1806be8   80   403   403 0000100 [SLPQ ufs 0xc1526964][SLP] 
> httpd
>  3056 c1994be8   80   403   403 0000100 [SLPQ ufs 0xc17e9400][SLP] 
> httpd
>  3055 c1806000   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3053 c137dde4   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3051 c19b05f4   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3048 c19941fc   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  3047 c19b03f8   80   403   403 0000100 [SLPQ suspfs 0xc137086c][SLP]
> httpd
>  3045 c1995be8   80   403   403 0000100 [SLPQ suspfs 0xc137086c][SLP]
> httpd
>  3036 c19945f4   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  2993 c162c1fc 1000  2992  2987 0004102 [SLPQ ufs 0xc13c4a78][SLP]
> mksnap_ffs
>  2992 c162cbe8 1000  2990  2987 0004002 [SLPQ wait 0xc162cbe8][SLP] sh
>  2991 c1995de4 1000  2987  2987 0004002 [SLPQ piperd 0xc13a9780][SLP] 
> gzip
>  2990 c150f5f4 1000  2987  2987 0004002 [SLPQ wait 0xc150f5f4][SLP] 
> dump
>  2987 c1806de4 1000  1762  2987 0004002 [SLPQ pause 0xc1806e18][SLP] 
> csh
>  2910 c1994de4   80   403   403 0000100 [SLPQ wait 0xc1994de4][SLP] 
> httpd
>  2909 c18053f8   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  2908 c19ae3f8   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  2843 c1995000   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  2839 c19955f4   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  2669 c1629000   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  2609 c18069ec   80   403   403 0000100 [SLPQ suspfs 0xc137086c][SLP]
> httpd
>  2598 c19943f8   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  1981 c18051fc   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  1899 c16295f4   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>  1762 c162cde4 1000  1761  1762 0004002 [SLPQ pause 0xc162ce18][SLP] 
> tcsh
>  1761 c162c000 1000  1759  1759 0000100 [SLPQ select 0xc078a744][SLP] 
> sshd
>  1759 c1629be8    0   368  1759 0004100 [SLPQ sbwait 0xc1429480][SLP] 
> sshd
>  1754 c18065f4   80   403   403 0000100 [SLPQ ufs 0xc13c4a78][SLP] 
> httpd
>   526 c162c9ec 1062   516   526 0004100 [SLPQ piperd 0xc13a9900][SLP]
> sendmail
>   525 c162c7f0 1062     1   525 0004000 [SLPQ accept 0xc1429186][SLP] 
> perl
>   516 c18057f0    0   390   390 0000000 [SLPQ piperd 0xc13a9c00][SLP] 
> cron
>   447 c12ab5f4    0     1   447 0004002 [SLPQ ttyin 0xc1318410][SLP] 
> getty
>   446 c15107f0    0     1   446 0004002 [SLPQ ttyin 0xc12f5810][SLP] 
> getty
>   445 c15105f4    0     1   445 0004002 [SLPQ ttyin 0xc12f6810][SLP] 
> getty
>   444 c15101fc    0     1   444 0004002 [SLPQ ttyin 0xc12f6410][SLP] 
> getty
>   443 c150f9ec    0     1   443 0004002 [SLPQ ttyin 0xc12ef410][SLP] 
> getty
>   442 c15103f8    0     1   442 0004002 [SLPQ ttyin 0xc12ef810][SLP] 
> getty
>   441 c150f3f8    0     1   441 0004002 [SLPQ ttyin 0xc12efc10][SLP] 
> getty
>   440 c150f1fc    0     1   440 0004002 [SLPQ ttyin 0xc12f5010][SLP] 
> getty
>   439 c150fde4    0     1   439 0004002 [SLPQ ttyin 0xc12f5410][SLP] 
> getty
>   427 c150fbe8    0     1   427 0000000 [SLPQ select 0xc078a744][SLP]
> inetd
>   403 c150f7f0    0     1   403 0000000 [SLPQ select 0xc078a744][SLP]
> httpd
>   390 c12ab9ec    0     1   390 0000000 [SLPQ nanslp 0xc0785dac][SLP] 
> cron
>   378 c137d000   25     1   378 0000100 [SLPQ pause 0xc137d034][SLP]
> sendmail
>   374 c12abbe8    0     1   374 0000100 [SLPQ select 0xc078a744][SLP]
> sendmail
>   368 c137d1fc    0     1   368 0000100 [SLPQ select 0xc078a744][SLP] 
> sshd
>   252 c12ab3f8    0     1   252 0000000 [SLPQ select 0xc078a744][SLP]
> syslogd
>   227 c12abde4    0     1   227 0000000 [SLPQ select 0xc078a744][SLP] 
> devd
>    46 c137d5f4    0     0     0 0000204 [SLPQ - 0xcb30cd14][SLP] 
> schedcpu
>    45 c137d7f0    0     0     0 0000204 [SLPQ - 0xc07923ec][SLP] 
> nfsiod 3
>    44 c137d9ec    0     0     0 0000204 [SLPQ - 0xc07923e8][SLP] 
> nfsiod 2
>    43 c137dbe8    0     0     0 0000204 [SLPQ - 0xc07923e4][SLP] 
> nfsiod 1
>    42 c1243be8    0     0     0 0000204 [SLPQ - 0xc07923e0][SLP] 
> nfsiod 0
>    41 c1243de4    0     0     0 0000204 [SLPQ vlruwt 0xc1243de4][SLP]
> vnlru
>    40 c12a8000    0     0     0 0000204 [SLPQ syncer 0xc0785b0c][SLP]
> syncer
>    39 c12a81fc    0     0     0 0000204 [SLPQ psleep 0xc078ac90][SLP]
> bufdaemon
>    38 c12a83f8    0     0     0 000020c [SLPQ pgzero 0xc07989e4][SLP]
> pagezero
>     9 c12a85f4    0     0     0 0000204 [SLPQ psleep 0xc0798534][SLP]
> vmdaemon
>     8 c12a87f0    0     0     0 0000204 [SLPQ psleep 0xc07984f0][SLP]
> pagedaemon
>    37 c12a89ec    0     0     0 0000204 [IWAIT] swi0: sio
>     7 c12a8be8    0     0     0 0000204 [SLPQ - 0xc120983c][SLP] fdc0
>    36 c12a8de4    0     0     0 0000204 [SLPQ usbtsk 0xc0780814][SLP]
> usbtask
>    35 c12ab000    0     0     0 0000204 [SLPQ usbevt 0xc12bb210][SLP] 
> usb0
>    34 c12ab1fc    0     0     0 0000204 [IWAIT] swi5:+
>     6 c12335f4    0     0     0 0000204 [SLPQ - 0xc1263dc0][SLP] thread
> taskq
>    33 c12337f0    0     0     0 0000204 [IWAIT] swi6:+
>    32 c12339ec    0     0     0 0000204 [IWAIT] swi6: task queue
>     5 c1233be8    0     0     0 0000204 [SLPQ - 0xc128e040][SLP] kqueue
> taskq
>    31 c1233de4    0     0     0 0000204 [IWAIT] swi2: cambio
>    30 c1243000    0     0     0 0000204 [SLPQ - 0xc077e7a0][SLP] yarrow
>     4 c12431fc    0     0     0 0000204 [SLPQ - 0xc0783148][SLP] g_down
>     3 c12433f8    0     0     0 0000204 [SLPQ - 0xc0783144][SLP] g_up
>     2 c12435f4    0     0     0 0000204 [SLPQ - 0xc078313c][SLP] 
> g_event
>    29 c12437f0    0     0     0 0000204 [IWAIT] swi3: vm
>    28 c12439ec    0     0     0 000020c [IWAIT] swi4: clock sio
>    27 c12191fc    0     0     0 0000204 [IWAIT] swi1: net
>    26 c12193f8    0     0     0 0000204 [IWAIT] irq15: ata1
>    25 c12195f4    0     0     0 0000204 [IWAIT] irq14: ata0
>    24 c12197f0    0     0     0 0000204 [IWAIT] irq13:
>    23 c12199ec    0     0     0 0000204 [IWAIT] irq12:
>    22 c1219be8    0     0     0 0000204 [IWAIT] irq11:
>    21 c1219de4    0     0     0 0000204 [IWAIT] irq10:
>    20 c1233000    0     0     0 0000204 [IWAIT] irq9: twe0
>    19 c12331fc    0     0     0 0000204 [IWAIT] irq8: rtc
>    18 c12333f8    0     0     0 0000204 [IWAIT] irq7: ppc0
>    17 c1211000    0     0     0 0000204 [IWAIT] irq6: fdc0
>    16 c12111fc    0     0     0 0000204 [IWAIT] irq5: bge0
>    15 c12113f8    0     0     0 0000204 [IWAIT] irq4: sio0
>    14 c12115f4    0     0     0 0000204 [IWAIT] irq3: uhci0
>    13 c12117f0    0     0     0 0000204 [IWAIT] irq1: atkbd0
>    12 c12119ec    0     0     0 0000204 [IWAIT] irq0: clk
>    11 c1211be8    0     0     0 000020c [CPU 0] idle
>     1 c1211de4    0     0     1 0004200 [SLPQ wait 0xc1211de4][SLP] 
> init
>    10 c1219000    0     0     0 0000204 [SLPQ ktrace 0xc0783d38][SLP]
> ktrace
>     0 c07832a0    0     0     0 0000200 [SLPQ sched 0xc07832a0][SLP]
> swapper
>   517 c162c5f4 1062   516   517 0006000 zomb[INACTIVE] sh
> db> show pcpu
> cpuid        = 0
> curthread    = 0xc12125c0: pid 11 "idle"
> curpcb       = 0xc69b7da0
> fpcurthread  = none
> idlethread   = 0xc12125c0: pid 11 "idle"
> APIC ID      = 0
> currentldt   = 0x28
>
> Joe 'sh' process in ufs:
>
> db> trace 3296
> Tracing pid 3296 tid 100255 td 0xc1cfa730
> sched_switch(c1cfa730,0,1) at sched_switch+0x17f
> mi_switch(1,0) at mi_switch+0x264
> sleepq_switch(c13c4a78,cbad385c,c0526901,c13c4a78,0) at 
> sleepq_switch+0xe0
> sleepq_wait(c13c4a78,0,0,c13c4a78,1010040) at sleepq_wait+0x30
> msleep(c13c4a78,c0784458,50,c06f08fe,0) at msleep+0x2d1
> acquire(cbad38b4,1010040,600,c1cfa730,0) at acquire+0x9a
> lockmgr(c13c4a78,1010002,c13c49b4,c1cfa730,cbad38d4) at lockmgr+0x382
> vop_stdlock(cbad3904) at vop_stdlock+0x1b
> VOP_LOCK_APV(c076f340,cbad3904,cbad38f4,c06c279e,cbad3904) at
> VOP_LOCK_APV+0x7e
> ffs_lock(cbad3904) at ffs_lock+0x3b
> VOP_LOCK_APV(c076ed60,cbad3904) at VOP_LOCK_APV+0x7e
> vn_lock(c13c49b4,10002,c1cfa730,2,c136a800) at vn_lock+0xc2
> vget(c13c49b4,10002,c1cfa730,c1cfa730,c13c4bdc) at vget+0xc9
> ufs_ihashget(c136a800,2,2,cbad3a00,c06e98f4) at ufs_ihashget+0x98
> ffs_vget(c1370800,2,2,cbad3a00) at ffs_vget+0x37
> ufs_root(c1370800,cbad3a3c,c1cfa730,1,1) at ufs_root+0x18
> lookup(cbad3c34,c05197b6,1,c1cfa730,c06ea9fd) at lookup+0x593
> namei(cbad3c34,c05197b6,c1cf9460,1,0) at namei+0x34e
> do_execve(c1cfa730,cbad3cc4,0,0,cbad3cc4) at do_execve+0x15d
> kern_execve(c1cfa730,cbad3cc4,0,c6295000,c6295000) at kern_execve+0x74
> execve(c1cfa730,cbad3d14,3,0,282) at execve+0x32
> syscall(2f,2f,2f,8067058,806708c) at syscall+0x213
> Xint0x80_syscall() at Xint0x80_syscall+0x1f
> --- syscall (59, FreeBSD ELF32, execve), eip = 0x28121d5b, esp =
> 0xbfbfec1c, ebp = 0xbfbfec48 ---
>
> httpd holding a VV_ROOT vnode lock, blocked in ufs:
>
> db> trace 3057
> Tracing pid 3057 tid 100107 td 0xc1807a10
> sched_switch(c1807a10,0,1) at sched_switch+0x17f
> mi_switch(1,0) at mi_switch+0x264
> sleepq_switch(c1526964,cb707994,c0526901,c1526964,0) at 
> sleepq_switch+0xe0
> sleepq_wait(c1526964,0,0,c1526964,1010040) at sleepq_wait+0x30
> msleep(c1526964,c0785244,50,c06f08fe,0) at msleep+0x2d1
> acquire(cb7079ec,1010040,600,c1807a10,0) at acquire+0x9a
> lockmgr(c1526964,1010002,c15268a0,c1807a10,cb707a0c) at lockmgr+0x382
> vop_stdlock(cb707a3c) at vop_stdlock+0x1b
> VOP_LOCK_APV(c076f340,cb707a3c,cb707a2c,c06c279e,cb707a3c) at
> VOP_LOCK_APV+0x7e
> ffs_lock(cb707a3c) at ffs_lock+0x3b
> VOP_LOCK_APV(c076ed60,cb707a3c) at VOP_LOCK_APV+0x7e
> vn_lock(c15268a0,10002,c1807a10,cb707b24,cb707c38) at vn_lock+0xc2
> vget(c15268a0,2,c1807a10,1b8,c1807a10) at vget+0xc9
> vfs_cache_lookup(cb707b24) at vfs_cache_lookup+0x1bd
> VOP_LOOKUP_APV(c076ed60,cb707b24) at VOP_LOOKUP_APV+0x7e
> lookup(cb707c10,cb707b70,1,c1807a10,1) at lookup+0x359
> namei(cb707c10,cb707bc4,c0672ebb,c19cbad0,c0703dc4) at namei+0x34e
> kern_stat(c1807a10,81541f8,0,cb707c84) at kern_stat+0x35
> stat(c1807a10,cb707d14,2,0,296) at stat+0x1b
> syscall(280b002f,2f,bfbf002f,81541f8,8170) at syscall+0x213
> Xint0x80_syscall() at Xint0x80_syscall+0x1f
> --- syscall (188, FreeBSD ELF32, stat), eip = 0x2839359b, esp =
> 0xbfbfe9dc, ebp = 0xbfbfea78 ---
>
> download.cgi blocked in ufs holding a directory vnode lock:
>
> db> trace 3062
> Tracing pid 3062 tid 100050 td 0xc137e000
> sched_switch(c137e000,0,1) at sched_switch+0x17f
> mi_switch(1,0) at mi_switch+0x264
> sleepq_switch(c163bca0,cb2e5844,c0526901,c163bca0,0) at 
> sleepq_switch+0xe0
> sleepq_wait(c163bca0,0,0,c163bca0,1010040) at sleepq_wait+0x30
> msleep(c163bca0,c0784578,50,c06f08fe,0) at msleep+0x2d1
> acquire(cb2e589c,1010040,600,c137e000,0) at acquire+0x9a
> lockmgr(c163bca0,1010002,c163bbdc,c137e000,cb2e58bc) at lockmgr+0x382
> vop_stdlock(cb2e58ec) at vop_stdlock+0x1b
> VOP_LOCK_APV(c076f340,cb2e58ec,cb2e58dc,c06c279e,cb2e58ec) at
> VOP_LOCK_APV+0x7e
> ffs_lock(cb2e58ec) at ffs_lock+0x3b
> VOP_LOCK_APV(c076ed60,cb2e58ec) at VOP_LOCK_APV+0x7e
> vn_lock(c163bbdc,10002,c137e000,cb2e59d4,cb2e5c04) at vn_lock+0xc2
> vget(c163bbdc,2,c137e000,1e0,c137e000) at vget+0xc9
> vfs_cache_lookup(cb2e59d4) at vfs_cache_lookup+0x1bd
> VOP_LOOKUP_APV(c076ed60,cb2e59d4) at VOP_LOOKUP_APV+0x7e
> lookup(cb2e5bdc,cb2e5a76,1,c137e000,cb2e5a80) at lookup+0x359
> namei(cb2e5bdc,0,d7ec0005,c137e000,c050f370) at namei+0x34e
> vn_open_cred(cb2e5bdc,cb2e5cdc,1a4,c19ace00,3) at vn_open_cred+0x27e
> vn_open(cb2e5bdc,cb2e5cdc,1a4,3,c07849f8) at vn_open+0x1e
> kern_open(c137e000,804a794,0,1,1b6) at kern_open+0xb6
> open(c137e000,cb2e5d14,3,1,292) at open+0x1a
> syscall(2f,2f,2f,4,2814e280) at syscall+0x213
> Xint0x80_syscall() at Xint0x80_syscall+0x1f
> --- syscall (5, FreeBSD ELF32, open), eip = 0x280ba23b, esp = 
> 0xbfbfe10c,
> ebp = 0xbfbfe138 ---
>
> httpd holding a directory lock blocked in ufs:
>
> db> trace 3056
> Tracing pid 3056 tid 100099 td 0xc162dcf0
> sched_switch(c162dcf0,0,1) at sched_switch+0x17f
> mi_switch(1,0) at mi_switch+0x264
> sleepq_switch(c17e9400,cb6be994,c0526901,c17e9400,0) at 
> sleepq_switch+0xe0
> sleepq_wait(c17e9400,0,0,c17e9400,1010040) at sleepq_wait+0x30
> msleep(c17e9400,c0784c38,50,c06f08fe,0) at msleep+0x2d1
> acquire(cb6be9ec,1010040,600,c162dcf0,0) at acquire+0x9a
> lockmgr(c17e9400,1010002,c17e933c,c162dcf0,cb6bea0c) at lockmgr+0x382
> vop_stdlock(cb6bea3c) at vop_stdlock+0x1b
> VOP_LOCK_APV(c076f340,cb6bea3c,cb6bea2c,c06c279e,cb6bea3c) at
> VOP_LOCK_APV+0x7e
> ffs_lock(cb6bea3c) at ffs_lock+0x3b
> VOP_LOCK_APV(c076ed60,cb6bea3c) at VOP_LOCK_APV+0x7e
> vn_lock(c17e933c,10002,c162dcf0,cb6beb24,cb6bec38) at vn_lock+0xc2
> vget(c17e933c,2,c162dcf0,210,c162dcf0) at vget+0xc9
> vfs_cache_lookup(cb6beb24) at vfs_cache_lookup+0x1bd
> VOP_LOOKUP_APV(c076ed60,cb6beb24) at VOP_LOOKUP_APV+0x7e
> lookup(cb6bec10,c17e9450,1,c162dcf0,75c) at lookup+0x359
> namei(cb6bec10,c05197b6,c0798060,1,c06ea9fd) at namei+0x34e
> kern_stat(c162dcf0,8161ad8,0,cb6bec84) at kern_stat+0x35
> stat(c162dcf0,cb6bed14,2,0,292) at stat+0x1b
> syscall(280b002f,2f,bfbf002f,8161ad8,8170) at syscall+0x213
> Xint0x80_syscall() at Xint0x80_syscall+0x1f
> --- syscall (188, FreeBSD ELF32, stat), eip = 0x2839359b, esp =
> 0xbfbfe75c, ebp = 0xbfbfe7f8 ---
>
> httpd holding a file lock, blocked in suspfs from ufs_inactive:
>
> db> trace 3045
> Tracing pid 3045 tid 100123 td 0xc1996450
> sched_switch(c1996450,0,1) at sched_switch+0x17f
> mi_switch(1,0) at mi_switch+0x264
> sleepq_switch(c137086c,cb6ecb8c,c0526901,c137086c,0) at 
> sleepq_switch+0xe0
> sleepq_wait(c137086c,0,0,1,c153be38) at sleepq_wait+0x30
> msleep(c137086c,c1370844,29f,c06f2e66,0) at msleep+0x2d1
> vn_write_suspend_wait(c17e933c,c1370800,1,0,c06f2634) at
> vn_write_suspend_wait+0xe4
> ufs_inactive(cb6ecc14) at ufs_inactive+0x186
> VOP_INACTIVE_APV(c076ed60,cb6ecc14) at VOP_INACTIVE_APV+0x7e
> vrele(c17e933c,c17e7210,0,c07046ce,19d) at vrele+0x10e
> vm_object_vndeallocate(c17e7210) at vm_object_vndeallocate+0xbd
> vm_object_deallocate(c17e7210,c17e7210,0,c0703dc4,89e) at
> vm_object_deallocate+0x85
> vm_map_entry_delete(c19cbbb8,c16258c4,284c8000,c19cbbb8,c16258c4) at
> vm_map_entry_delete+0x130
> vm_map_delete(c19cbbb8,284c8000,284c9000,0,c1996450) at
> vm_map_delete+0x18f
> munmap(c1996450,cb6ecd14,2,3,292) at munmap+0xa3
> syscall(280b002f,2f,bfbf002f,2831e058,815f2d8) at syscall+0x213
> Xint0x80_syscall() at Xint0x80_syscall+0x1f
> --- syscall (73, FreeBSD ELF32, munmap), eip = 0x28393cbb, esp =
> 0xbfbfc6fc, ebp = 0xbfbfc718 ---
>
> Another httpd holding a file lock, blocked in suspfs from ufs_inactive:
>
> db> trace 3047
> Tracing pid 3047 tid 100143 td 0xc19b2450
> sched_switch(c19b2450,0,1) at sched_switch+0x17f
> mi_switch(1,0) at mi_switch+0x264
> sleepq_switch(c137086c,cb781b8c,c0526901,c137086c,0) at 
> sleepq_switch+0xe0
> sleepq_wait(c137086c,0,0,1,c141ec08) at sleepq_wait+0x30
> msleep(c137086c,c1370844,29f,c06f2e66,0) at msleep+0x2d1
> vn_write_suspend_wait(c18419b4,c1370800,1,0,c06f2634) at
> vn_write_suspend_wait+0xe4
> ufs_inactive(cb781c14) at ufs_inactive+0x186
> VOP_INACTIVE_APV(c076ed60,cb781c14) at VOP_INACTIVE_APV+0x7e
> vrele(c18419b4,c1826000,0,c07046ce,19d) at vrele+0x10e
> vm_object_vndeallocate(c1826000) at vm_object_vndeallocate+0xbd
> vm_object_deallocate(c1826000,c1826000,0,c0703dc4,89e) at
> vm_object_deallocate+0x85
> vm_map_entry_delete(c19cbe10,c18bfd8c,284cd000,c19cbe10,c18bfd8c) at
> vm_map_entry_delete+0x130
> vm_map_delete(c19cbe10,284cd000,284ce000,0,c19b2450) at
> vm_map_delete+0x18f
> munmap(c19b2450,cb781d14,2,3,292) at munmap+0xa3
> syscall(280b002f,2f,bfbf002f,2831e058,815d228) at syscall+0x213
> Xint0x80_syscall() at Xint0x80_syscall+0x1f
> --- syscall (73, FreeBSD ELF32, munmap), eip = 0x28393cbb, esp =
> 0xbfbfc6fc, ebp = 0xbfbfc718 ---
>
> mksnap_ffs, creating the snapshot, but blocked on a vnode lock:
>
> db> trace 2993
> Tracing pid 2993 tid 100096 td 0xc162d8a0
> sched_switch(c162d8a0,0,1) at sched_switch+0x17f
> mi_switch(1,0) at mi_switch+0x264
> sleepq_switch(c13c4a78,cb6b560c,c0526901,c13c4a78,0) at 
> sleepq_switch+0xe0
> sleepq_wait(c13c4a78,0,0,c13c4a78,1010040) at sleepq_wait+0x30
> msleep(c13c4a78,c0784458,50,c06f08fe,0) at msleep+0x2d1
> acquire(cb6b5664,1010040,600,c162d8a0,0) at acquire+0x9a
> lockmgr(c13c4a78,1010002,c13c49b4,c162d8a0,cb6b5684) at lockmgr+0x382
> vop_stdlock(cb6b56b4) at vop_stdlock+0x1b
> VOP_LOCK_APV(c076f340,cb6b56b4,cb6b56a4,c06c279e,cb6b56b4) at
> VOP_LOCK_APV+0x7e
> ffs_lock(cb6b56b4) at ffs_lock+0x3b
> VOP_LOCK_APV(c076ed60,cb6b56b4) at VOP_LOCK_APV+0x7e
> vn_lock(c13c49b4,10002,c162d8a0) at vn_lock+0xc2
> ffs_snapshot(c1370800,c14ccbc0,c14ccbc0,c1369b00,0) at 
> ffs_snapshot+0x1242
> ffs_mount(c1370800,c162d8a0,20000000,201000,c13c49b4) at 
> ffs_mount+0x7a5
> vfs_domount(c162d8a0,c13207b0,c141c030,1211000,c1320670) at
> vfs_domount+0x576
> vfs_donmount(c162d8a0,1211000,cb6b5c14,c1422880,e) at vfs_donmount+0xce
> kernel_mount(c1320690,1211000,bfbfed33,0,0) at kernel_mount+0x6d
> ffs_cmount(c1320690,bfbfe578,1211000,c162d8a0,c076ea80) at 
> ffs_cmount+0x5d
> mount(c162d8a0,cb6b5d14,4,2,206) at mount+0x156
> syscall(2f,2f,2f,bfbfe7c8,bfbfed29) at syscall+0x213
> Xint0x80_syscall() at Xint0x80_syscall+0x1f
> --- syscall (21, FreeBSD ELF32, mount), eip = 0x280b80db, esp =
> 0xbfbfe4ec, ebp = 0xbfbfebe0 ---
>
> Anothr httpd blocked in suspfs while holding a file vnode lock, trying 
> to
> unmap:
>
> db>  trace 2609
> Tracing pid 2609 tid 100108 td 0xc1807b80
> sched_switch(c1807b80,0,1) at sched_switch+0x17f
> mi_switch(1,0) at mi_switch+0x264
> sleepq_switch(c137086c,cb70ab8c,c0526901,c137086c,0) at 
> sleepq_switch+0xe0
> sleepq_wait(c137086c,0,0,1,c14cae38) at sleepq_wait+0x30
> msleep(c137086c,c1370844,29f,c06f2e66,0) at msleep+0x2d1
> vn_write_suspend_wait(c19fd9b4,c1370800,1,0,c06f2634) at
> vn_write_suspend_wait+0xe4
> ufs_inactive(cb70ac14) at ufs_inactive+0x186
> VOP_INACTIVE_APV(c076ed60,cb70ac14) at VOP_INACTIVE_APV+0x7e
> vrele(c19fd9b4,c1a93948,0,c07046ce,19d) at vrele+0x10e
> vm_object_vndeallocate(c1a93948) at vm_object_vndeallocate+0xbd
> vm_object_deallocate(c1a93948,c1a93948,0,c0703dc4,89e) at
> vm_object_deallocate+0x85
> vm_map_entry_delete(c18bd834,c18cb3b8,284c7000,c18bd834,c18cb3b8) at
> vm_map_entry_delete+0x130
> vm_map_delete(c18bd834,284c7000,284c8000,0,c1807b80) at
> vm_map_delete+0x18f
> munmap(c1807b80,cb70ad14,2,37,296) at munmap+0xa3
> syscall(2f,2f,bfbf002f,2831e058,815d0c8) at syscall+0x213
> Xint0x80_syscall() at Xint0x80_syscall+0x1f
> --- syscall (73, FreeBSD ELF32, munmap), eip = 0x28393cbb, esp =
> 0xbfbfc5cc, ebp = 0xbfbfc5e8 ---
>
> No WITNESS in this kernel.
>
> My first glance reading is that there's a deadlock between the snapshot
> lock and vnode locks, which ocurs because a vnode lock has been 
> acquired
> as part of the mmap tear-down, which leads to vn_write_suspend() being
> called in ufs_inactive as the inode has a zero reference count so has 
> to
> be GC'd, which involves writing to the file system.  VOP_INACTIVE is
> called by vrele() with the vnode lock held.  My understanding is that,
> generally, we acquire the file system snapshot lock before vnode locks 
> as
> part of the VFS lock order.
>
> This could be a misinterpretation.  I'll try to get a dump, but this 
> is a
> twe box so I may not be able to.
>
> Robert N M Watson
>
>
> _______________________________________________
> freebsd-current_at_freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-current
> To unsubscribe, send any mail to 
> "freebsd-current-unsubscribe_at_freebsd.org"
Received on Sat Feb 19 2005 - 16:51:31 UTC

This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:38:28 UTC