Problem remains with FreeBSD 6.0-RC1 as seen in RELENG_5

From: Philip Kizer <pckizer_at_nostrum.com>
Date: Wed, 19 Oct 2005 11:23:27 -0500
I have a problem I reported on freebsd-stable several weeks ago in:

  http://www.FreeBSD.org/cgi/getmsg.cgi?fetch=47770+187449+/usr/local/www/db/text/2005/freebsd-stable/20051009.freebsd-stable


I upgraded a test box to see if all of the reports were true that threading
and most other major problems were better in the 6.x branch, but I have had
the same kind of hangs with 6.0-RC1 that I was having with RELENG_5.

I get notified that some of my services are unavailable and I verify that
new connection attempts from remote just hang.  Attempts at issuing
commands from my existing ssh connections will let me send a <return> and
see a new prompt generated, but any attempt at execing/etc will then hang
that process.


Moving to the console, I get the same behaviour such that if it is already
logged in, I can hit return in the shell and get a new shell prompt, but
any command I try (i.e.  'uptime' or 'uname') then results in the same lack
of any further response.  [Though, obviously from below, sending a break to
activate DDB still works.]

If the console is not already logged in and I hit return at "login:", I get
a new "login:" prompt; but, as soon as I enter a username+^M or even Ctrl-D
it, I cease to receive any feedback besides terminal echo from my input.


This is:
FreeBSD [host-elided] 6.0-RC1 FreeBSD 6.0-RC1 #0: Mon Oct 17 15:52:35 CDT 2005     root_at_[host-elided]:/usr/obj/usr/src/sys/SMP_DDB  i386

# grep '^[a-z]' /sys/i386/conf/SMP_DDB
include GENERIC
ident           DDB
options         SMP                     # Symmetric MultiProcessor Kernel
options         INCLUDE_CONFIG_FILE     # Include this file in kernel
options DDB                     #Enable the kernel debugger
options DDB_NUMSYM              #Print numerical value of symbols too
options KDB
options KDB_TRACE
options BREAK_TO_DEBUGGER

[After notification of a problem, this is from the serial console...a few
further inline comments included in brackets "[]".]

#
[Typed: Ctrl-T]
load: 1.82  cmd: sh 795 [ttyin] 0.00u 0.00s 0% 168k
# uptime
[Waited a while to be sure it was hung and not returning. Retyped: Ctrl-T]
load: 1.82  cmd: uptime 83303 [filelist lock] 0.00u 0.00s 0% 180k
[Sent BREAK]
KDB: enter: Line break on console
[thread pid 82885 tid 100133 ]
Stopped at	0xc0659b37 = kdb_enter+0x2b:	nop
db> set $lines=0
db> trace
Tracing pid 82885 tid 100133 td 0xc3c34c00
kdb_enter(c0895a5e) at 0xc0659b37 = kdb_enter+0x2b
siointr1(c372ec00) at 0xc07f09b2 = siointr1+0xce
siointr(c372ec00) at 0xc07f0792 = siointr+0x5e
intr_execute_handlers(c3528090,f1325b68,4,f1325bcc,c0805073) at 0xc0808985 = intr_execute_handlers+0x85
lapic_handle_intr(34) at 0xc080b042 = lapic_handle_intr+0x2e
Xapic_isr1() at 0xc0805073 = Xapic_isr1+0x33
--- interrupt, eip = 0xc0685d59, esp = 0xf1325bac, ebp = 0xf1325bcc ---
unp_gc(c43be2c8) at 0xc0685d59 = unp_gc+0xf1
unp_detach(c43b7000,f1325c08,c067a188,c43be2c8,0) at 0xc06840fa = unp_detach+0xae
uipc_detach(c43be2c8) at 0xc0682f8e = uipc_detach+0x82
soclose(c43be2c8,c40fb6c0,0,f1325c34,c0620c10) at 0xc067a188 = soclose+0x278
soo_close(c40fb6c0,c3c34c00) at 0xc0669fb3 = soo_close+0x63
fdrop_locked(c40fb6c0,c3c34c00,c3a3d800,f1325cb4,c061f0fb) at 0xc0620c10 = fdrop_locked+0xac
fdrop(c40fb6c0,c3c34c00,c066371f,c40fb6c0,c878ac00) at 0xc0620b5d = fdrop+0x41
closef(c40fb6c0,c3c34c00,0,c3c34c00,6) at 0xc061f0fb = closef+0x427
close(c3c34c00,f1325d04,1,0,292) at 0xc061c4bd = close+0x211
syscall(3b,1003b,bfbf003b,80d58e7,80e8780) at 0xc0818557 = syscall+0x2bf
Xint0x80_syscall() at 0xc0804d0f = Xint0x80_syscall+0x1f
--- syscall (6, FreeBSD ELF32, close), eip = 0x282cb7d3, esp = 0xbfbfae4c, ebp = 0xbfbfae68 ---
db> show reg
cs		  0x20
ds	    0xc0870028
es	    0xc0650028	parse_dynamic+0x4
fs	    0xf1320008
ss		  0x28
eax		  0x22
ecx	    0xc1033000
edx		     0
ebx	    0xc3c34c00
esp	    0xf1325afc
ebp	    0xf1325afc
esi	    0xc372ec00
edi		  0xf9
eip	    0xc0659b37	kdb_enter+0x2b
efl		  0x86
dr0		     0
dr1		     0
dr2		     0
dr3		     0
dr4	    0xffff0ff0
dr5		 0x400
dr6	    0xffff0ff0
dr7		 0x400
0xc0659b37 = kdb_enter+0x2b:	nop
db> show witness
No such command
db> show pcpu
cpuid	     = 1
curthread    = 0xc3c34c00: pid 82885 \"sendmail\"
curpcb	     = 0xf1325d90
fpcurthread  = none
idlethread   = 0xc34d8780: pid 11 \"idle: cpu1\"
APIC ID      = 0
currentldt   = 0x50
db> show pcpuu0
cpuid	     = 0
curthread    = 0xc34d8900: pid 12 \"idle: cpu0\"
curpcb	     = 0xe4f9bd90
fpcurthread  = none
idlethread   = 0xc34d8900: pid 12 \"idle: cpu0\"
APIC ID      = 1
currentldt   = 0x50
db> show pcpu 1
cpuid	     = 1
curthread    = 0xc3c34c00: pid 82885 \"sendmail\"
curpcb	     = 0xf1325d90
fpcurthread  = none
idlethread   = 0xc34d8780: pid 11 \"idle: cpu1\"
APIC ID      = 0
currentldt   = 0x50
db> show pcpu 2
CPU 2 not found
db> print sysctllock
c093ace0
db> ps
  pid	proc	 uid  ppid  pgrp  flag	 stat  wmesg	wchan  cmd
83305 c43b2830	  0   504   504 0000000 [SLPQ filelist lock 0xc0938bd0][SLP] cron
83304 c3d91a3c 21802   605   605 0000000 [SLPQ filelist lock 0xc0938bd0][SLP] perl
83303 c3d12c48	  0   795 83303 0004002 [SLPQ filelist lock 0xc0938bd0][SLP] uptime
83302 c5b13418	  0   686   686 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83301 c3ba1c48	  0   680   680 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83300 c40c320c	  0   504   504 0000000 [SLPQ filelist lock 0xc0938bd0][SLP] cron
83299 c43a2830	  0   504   504 0000000 [SLPQ filelist lock 0xc0938bd0][SLP] cron
83298 c38dec48	  0   504   504 0000000 [SLPQ filelist lock 0xc0938bd0][SLP] cron
83297 c3f0020c 21802   605   605 0000000 [SLPQ filelist lock 0xc0938bd0][SLP] perl
83296 c3c36624 21802   605   605 0000000 [SLPQ filelist lock 0xc0938bd0][SLP] perl
83295 c406620c 21802   605   605 0000000 [SLPQ filelist lock 0xc0938bd0][SLP] perl
83294 c38de20c 21802   605   605 0000000 [SLPQ filelist lock 0xc0938bd0][SLP] perl
83293 c41af418	  0   686   686 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83292 c4244a3c	  0   680   680 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83291 c40c0624	  0   504   504 0000000 [SLPQ filelist lock 0xc0938bd0][SLP] cron
83290 c43b1a3c	  0   504   504 0000000 [SLPQ filelist lock 0xc0938bd0][SLP] cron
83289 c5fffc48	  0   664   664 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83288 c3e9d830	  0   650   650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83287 c3cbbc48	  0   650   650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83286 c43a2000	  0   636   636 0000100 [SLPQ select 0xc0941aa4][SLP] sendmail
83285 c40c4c48	  0   650   650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83284 c413b624	  0   636   636 0000100 [SLPQ select 0xc0941aa4][SLP] sendmail
83283 c6003418	  0   664   664 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83282 c4140000	  0   650   650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83281 c43ac000	  0   650   650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83280 c3d91830	  0   650   650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83279 c43afc48	  0   664   664 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83278 c3cbb418	  0   664   664 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83277 c4244c48	  0   650   650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83276 c40c0830	  0   650   650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83275 c4136624	  0   664   664 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83274 c4140a3c	  0   650   650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83273 c6035830	  0   636   636 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83272 c43b120c	  0   664   664 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83271 c41ada3c	  0   650   650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83270 c3fca624	  0   664   664 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83269 c6035c48	  0   664   664 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83268 c5b13c48	  0   664   664 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83267 c3f82c48	  0   664   664 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83266 c424320c	  0   664   664 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83265 c43a720c	  0   636   636 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83264 c3e9d418	  0   650   650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83263 c3d18000	  0   650   650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83261 c6036418	  0   664   664 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83260 c5b0120c	  0   650   650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83259 c4244000	  0   650   650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83258 c4243830	  0   664   664 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83257 c43af830	  0	1 83257 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83255 c5b01830	  0   650   650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83254 c5b01000	  0   650   650 0000100 [SLPQ select 0xc0941aa4][SLP] sendmail
83248 c3b99c48	  0   680   680 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83247 c42d5c48	  0   650   650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83246 c3f00a3c	  0   664   664 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83245 c6036000	  0   650   650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83244 c40c3c48	  0   650   650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83243 c5fff830	  0   636   636 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83242 c5b01418	  0   650   650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83241 c42a2624	  0   650   650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83240 c41af000	  0   650   650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83239 c41ad830	  0   636   636 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83238 c3d1820c	  0   650   650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83237 c3f00624	  0   664   664 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83236 c5b13a3c	  0   650   650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83235 c42a7a3c	  0   664   664 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83234 c3f82830	  0   650   650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83233 c43a7624	  0   664   664 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83231 c4376418	  0   683   683 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83230 c406e830	  0   664   664 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83229 c42d520c	  0   650   650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83228 c43af418	  0   650   650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83227 c413ba3c	  0   664   664 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83226 c3e9d20c	  0   650   650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83225 c43a220c	  0   650   650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83224 c3b99418	  0   636   636 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83223 c3c33c48	  0   650   650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83222 c4375000	  0   650   650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83221 c3e9d624	  0   664   664 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83220 c38de830	  0   636   636 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83219 c3f8020c	  0   650   650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83218 c3f02a3c	  0   650   650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83213 c4375a3c	  0   650   650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83212 c5b09418	  0   650   650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83211 c43af624	  0   664   664 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83208 c4135830	  0   650   650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83207 c41afa3c	  0   650   650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83206 c406ea3c	  0   650   650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83205 c413b418	  0   664   664 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83195 c3f80624	  0   650   650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83193 c5b0920c	  0   664   664 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83192 c3c33624	  0   650   650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83191 c3c36a3c	  0   650   650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83190 c4140418	  0   636   636 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83189 c4135418	  0   650   650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83188 c437620c	  0   636   636 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83187 c406e624	  0   650   650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83186 c43a7418	  0   650   650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83185 c3c36c48	  0   650   650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83184 c3cbb830	  0   650   650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83183 c406e000	  0   650   650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83182 c43a2c48	  0   650   650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83181 c40c3000	  0   664   664 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83180 c4140c48	  0   664   664 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83179 c42a2418	  0   650   650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83178 c437e000	  0   664   664 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83175 c5fff20c	  0   650   650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83174 c3f82624	  0   650   650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83168 c3ba1830	  0   650   650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83167 c4243a3c	  0   664   664 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83164 c6003000	  0   650   650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83162 c4066a3c	  0   664   664 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83157 c40c4000	  0   664   664 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83154 c600320c	  0   650   650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83153 c40c0c48	  0   664   664 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83152 c3d18418	  0   650   650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83150 c41adc48	  0   650   650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83149 c3e9dc48	  0   650   650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83148 c3cba624	  0   664   664 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83146 c3c36000	  0   650   650 0000100 [SLPQ select 0xc0941aa4][SLP] sendmail
83145 c3f01830	  0   650   650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83142 c41af20c	  0   650   650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83141 c3e9da3c	  0   664   664 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83140 c5fff624	  0   650   650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83134 c41af624	  0   650   650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83132 c4136830	  0   650   650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83131 c603620c	  0   664   664 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83130 c3f00418	  0   650   650 0000100 [SLPQ select 0xc0941aa4][SLP] sendmail
83129 c3fcac48	  0   664   664 0000100 [SLPQ select 0xc0941aa4][SLP] sendmail
83128 c40c4418	  0   650   650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83125 c42d4c48	  0   664   664 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83124 c3fca20c	  0   650   650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83123 c437e418	  0   650   650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83121 c4135c48	  0   664   664 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83119 c41af830	  0   650   650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83118 c38dd000	  0   650   650 0000100 [SLPQ select 0xc0941aa4][SLP] sendmail
83117 c42d4a3c	  0   664   664 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83114 c42a720c	  0   650   650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83113 c41aac48	  0   650   650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83112 c42d420c	  0   664   664 0000100 [SLPQ select 0xc0941aa4][SLP] sendmail
83109 c40c4a3c	  0   650   650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83108 c3f8220c	  0   650   650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83107 c3d10830	  0   650   650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83105 c41aa000	  0   650   650 0000100 [SLPQ select 0xc0941aa4][SLP][SWAP] sendmail
83101 c4066624	  0   664   664 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83100 c40c3624	  0   650   650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83097 c42a2000	  0   650   650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83094 c41aa830	  0   650   650 0000100 [SLPQ select 0xc0941aa4][SLP] sendmail
83093 c43a7a3c	  0   664   664 0000100 [SLPQ select 0xc0941aa4][SLP] sendmail
83092 c43b1000	  0   650   650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83091 c437ec48	  0   650   650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83087 c5fff000	  0   650   650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83070 c42a2830	  0   650   650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83069 c6036c48	  0   650   650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83056 c6035624	  0   664   664 0000100 [SLPQ select 0xc0941aa4][SLP][SWAP] sendmail
83051 c41ad20c	  0   650   650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83050 c4244830	  0   664   664 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83041 c40c0a3c	  0   664   664 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83038 c424420c	  0   664   664 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
83037 c43a2624	  0   664   664 0000100 [SLPQ select 0xc0941aa4][SLP][SWAP] sendmail
83034 c413b000	  0   664   664 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
82994 c43b2000 21802   551   551 0000000 [SLPQ select 0xc0941aa4][SLP] perl
82993 c38d8c48 21802   550   550 0000000 [SLPQ select 0xc0941aa4][SLP] perl
82992 c5b01a3c 21802   551   551 0000000 [SLPQ select 0xc0941aa4][SLP] perl
82991 c5b01c48 21802   550   550 0000000 [SLPQ select 0xc0941aa4][SLP] perl
82990 c43b2624 21802   551   551 0000000 [SLPQ select 0xc0941aa4][SLP] perl
82989 c5b09000 21802   550   550 0000000 [SLPQ select 0xc0941aa4][SLP] perl
82987 c6036830 21802   551   551 0000000 [SLPQ select 0xc0941aa4][SLP] perl
82986 c42d5624 21802   550   550 0000000 [SLPQ select 0xc0941aa4][SLP] perl
82984 c5b09624 21802   551   551 0000000 [SLPQ select 0xc0941aa4][SLP] perl
82983 c5b09830 21802   550   550 0000000 [SLPQ select 0xc0941aa4][SLP] perl
82982 c4135a3c 21802   551   551 0000000 [SLPQ filelist lock 0xc0938bd0][SLP] perl
82981 c3f82000 21802   550   550 0000000 [SLPQ select 0xc0941aa4][SLP] perl
82976 c42d5830 21802   551   551 0000000 [SLPQ select 0xc0941aa4][SLP] perl
82975 c5b09c48 21802   550   550 0000000 [SLPQ select 0xc0941aa4][SLP] perl
82974 c43acc48 21802   551   551 0000000 [SLPQ select 0xc0941aa4][SLP] perl
82973 c3f02418 21802   550   550 0000000 [SLPQ select 0xc0941aa4][SLP] perl
82972 c5b13000 21802   551   551 0000000 [SLPQ select 0xc0941aa4][SLP] perl
82971 c43ac20c 21802   550   550 0000000 [SLPQ select 0xc0941aa4][SLP] perl
82970 c3cbb20c 21802   551   551 0000000 [SLPQ select 0xc0941aa4][SLP] perl
82969 c43a2418 21802   550   550 0000000 [SLPQ select 0xc0941aa4][SLP] perl
82967 c43a7000 21802   551   551 0000000 [SLPQ select 0xc0941aa4][SLP] perl
82966 c40c0000 21802   550   550 0000000 [SLPQ select 0xc0941aa4][SLP] perl
82965 c4376c48 21802   551   551 0000000 [SLPQ select 0xc0941aa4][SLP] perl
82964 c43b1c48 21802   550   550 0000000 [SLPQ select 0xc0941aa4][SLP] perl
82962 c41ad624 21802   551   551 0000000 [SLPQ select 0xc0941aa4][SLP] perl
82961 c38dea3c 21802   550   550 0000000 [SLPQ select 0xc0941aa4][SLP] perl
82960 c38a820c 21802   551   551 0000000 [SLPQ select 0xc0941aa4][SLP] perl
82959 c43b0c48 21802   550   550 0000000 [SLPQ select 0xc0941aa4][SLP] perl
82952 c42a2a3c 21802   551   551 0000000 [SLPQ select 0xc0941aa4][SLP] perl
82951 c6035000 21802   550   550 0000000 [SLPQ select 0xc0941aa4][SLP] perl
82948 c43b220c 21802   551   551 0000000 [SLPQ filelist lock 0xc0938bd0][SLP] perl
82947 c6035a3c 21802   550   550 0000000 [SLPQ select 0xc0941aa4][SLP] perl
82945 c43b2418 21802   551   551 0000000 [SLPQ select 0xc0941aa4][SLP] perl
82944 c41aaa3c 21802   550   550 0000000 [SLPQ select 0xc0941aa4][SLP] perl
82938 c40c420c 21802   551   551 0000000 [SLPQ select 0xc0941aa4][SLP] perl
82937 c6036a3c 21802   550   550 0000000 [SLPQ select 0xc0941aa4][SLP] perl
82932 c3f01a3c 21802   551   551 0000000 [SLPQ select 0xc0941aa4][SLP] perl
82931 c42d4624 21802   550   550 0000000 [SLPQ select 0xc0941aa4][SLP] perl
82930 c41aa418 21802   551   551 0000000 [SLPQ select 0xc0941aa4][SLP] perl
82929 c3d94c48 21802   550   550 0000000 [SLPQ select 0xc0941aa4][SLP] perl
82928 c3f02830 21802   551   551 0000000 [SLPQ select 0xc0941aa4][SLP] perl
82927 c3d94a3c 21802   550   550 0000000 [SLPQ select 0xc0941aa4][SLP] perl
82926 c413620c 21802   551   551 0000000 [SLPQ select 0xc0941aa4][SLP] perl
82925 c3fcbc48 21802   550   550 0000000 [SLPQ select 0xc0941aa4][SLP] perl
82924 c42d5a3c 21802   551   551 0000000 [SLPQ select 0xc0941aa4][SLP] perl
82923 c4140830 21802   550   550 0000000 [SLPQ select 0xc0941aa4][SLP] perl
82922 c43ac830 21802   551   551 0000000 [SLPQ select 0xc0941aa4][SLP] perl
82921 c3f0220c 21802   550   550 0000000 [SLPQ select 0xc0941aa4][SLP] perl
82919 c43b0418	  0   650   650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
82918 c3cbba3c 21802   551   551 0000000 [SLPQ select 0xc0941aa4][SLP] perl
82917 c4066830 21802   550   550 0000000 [SLPQ select 0xc0941aa4][SLP] perl
82916 c3ba1624	  0   636   636 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
82915 c4376000 21802   551   551 0000000 [SLPQ select 0xc0941aa4][SLP] perl
82914 c3f01c48 21802   550   550 0000000 [SLPQ select 0xc0941aa4][SLP] perl
82913 c43b1418	  0   636   636 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
82909 c4243c48 21802   551   551 0000000 [SLPQ select 0xc0941aa4][SLP] perl
82908 c3c33000 21802   550   550 0000000 [SLPQ select 0xc0941aa4][SLP] perl
82907 c4243000	  0   650   650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
82906 c437ea3c 21802   551   551 0000000 [SLPQ select 0xc0941aa4][SLP] perl
82905 c3ba1000 21802   550   550 0000000 [SLPQ select 0xc0941aa4][SLP] perl
82904 c43b1624	  0   650   650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
82903 c40c3a3c 21802   551   551 0000000 [SLPQ select 0xc0941aa4][SLP] perl
82902 c3f82a3c 21802   550   550 0000000 [SLPQ select 0xc0941aa4][SLP] perl
82896 c43b0830 21802   551   551 0000000 [SLPQ select 0xc0941aa4][SLP] perl
82895 c4136000 21802   550   550 0000000 [SLPQ select 0xc0941aa4][SLP] perl
82894 c42a7000 21802   551   551 0000000 [SLPQ select 0xc0941aa4][SLP] perl
82893 c3e9d000 21802   550   550 0000000 [SLPQ select 0xc0941aa4][SLP] perl
82892 c3c3320c 21802   551   551 0000000 [SLPQ select 0xc0941aa4][SLP] perl
82891 c42a220c 21802   550   550 0000000 [SLPQ select 0xc0941aa4][SLP] perl
82890 c4375c48	  0   650   650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
82889 c3ba1a3c	  0   664   664 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
82888 c43af000	  0   650   650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
82887 c42a2c48 21802   551   551 0000000 [SLPQ select 0xc0941aa4][SLP] perl
82886 c3c33418 21802   550   550 0000000 [SLPQ select 0xc0941aa4][SLP] perl
82885 c3c36830	  0   636   636 0000100 [CPU 1] sendmail
82877 c42a7418 21802   551   551 0000000 [SLPQ select 0xc0941aa4][SLP] perl
82876 c3cbaa3c 21802   550   550 0000000 [SLPQ select 0xc0941aa4][SLP] perl
82874 c413bc48	  0   650   650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
82870 c43a7c48	  0   650   650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
82865 c437520c 21802   551   551 0000000 [SLPQ select 0xc0941aa4][SLP] perl
82864 c4376830 21802   550   550 0000000 [SLPQ select 0xc0941aa4][SLP] perl
82851 c3f80418	  0   650   650 0000100 [SLPQ select 0xc0941aa4][SLP][SWAP] sendmail
82837 c3f02c48	  0   650   650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
82782 c42d5000 21802   551   551 0000000 [SLPQ select 0xc0941aa4][SLP] perl
82781 c3d9120c 21802   550   550 0000000 [SLPQ select 0xc0941aa4][SLP] perl
82769 c43aca3c	  0   650   650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
82766 c40c020c 21802   551   551 0000000 [SLPQ select 0xc0941aa4][SLP] perl
82765 c41aa20c 21802   550   550 0000000 [SLPQ select 0xc0941aa4][SLP] perl
82763 c3d1220c	  0   664   664 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
82762 c3d9420c 21802   551   551 0000000 [SLPQ select 0xc0941aa4][SLP] perl
82761 c42a7624 21802   550   550 0000000 [SLPQ select 0xc0941aa4][SLP] perl
82740 c6036624	  0   664   664 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
82738 c4136a3c 21802   551   551 0000000 [SLPQ select 0xc0941aa4][SLP] perl
82737 c4243418 21802   550   550 0000000 [SLPQ select 0xc0941aa4][SLP] perl
82729 c3f82418	  0   664   664 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
82676 c4376624	  0   664   664 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
82621 c3c33a3c	  0   650   650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
82607 c4376a3c	  0   650   650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
81310 c6035418	  0   664   664 0000100 [SLPQ select 0xc0941aa4][SLP][SWAP] sendmail
80076 c42d4000	  0   650   650 0000100 [SLPQ select 0xc0941aa4][SLP][SWAP] sendmail
79627 c4135624	  0   650   650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
79198 c3cba830 21802   551   551 0000000 [SLPQ select 0xc0941aa4][SLP] perl
79197 c414020c 21802   550   550 0000000 [SLPQ select 0xc0941aa4][SLP] perl
79193 c4066418 21802   551   551 0000000 [SLPQ filelist lock 0xc0938bd0][SLP] perl
79192 c40c3830 21802   550   550 0000000 [SLPQ select 0xc0941aa4][SLP] perl
79189 c3f00000 21802   551   551 0000000 [SLPQ select 0xc0941aa4][SLP] perl
79188 c437e830 21802   550   550 0000000 [SLPQ select 0xc0941aa4][SLP] perl
79186 c3d18830 21802   551   551 0000000 [SLPQ select 0xc0941aa4][SLP] perl
79185 c3fca418 21802   550   550 0000000 [SLPQ select 0xc0941aa4][SLP] perl
79184 c4136418 21802   551   551 0000000 [SLPQ select 0xc0941aa4][SLP] perl
79183 c3fcaa3c 21802   550   550 0000000 [SLPQ select 0xc0941aa4][SLP] perl
79179 c3f80000 21802   551   551 0000000 [SLPQ select 0xc0941aa4][SLP] perl
79178 c3fcb20c 21802   550   550 0000000 [SLPQ select 0xc0941aa4][SLP] perl
79168 c603520c 21802   551   551 0000000 [SLPQ filelist lock 0xc0938bd0][SLP] perl
79167 c4243624 21802   550   550 0000000 [SLPQ select 0xc0941aa4][SLP] perl
79083 c4375830	  0   664   664 0000100 [SLPQ sbwait 0xc4401900][SLP][SWAP] sendmail
78838 c3f01000 21802   551   551 0000000 [SLPQ select 0xc0941aa4][SLP] perl
78837 c3f80830 21802   550   550 0000000 [SLPQ select 0xc0941aa4][SLP] perl
78491 c40c4624 21802   551   551 0000000 [SLPQ select 0xc0941aa4][SLP] perl
78490 c3d94830 21802   550   550 0000000 [SLPQ select 0xc0941aa4][SLP] perl
78484 c3f02000 21802   551   551 0000000 [SLPQ select 0xc0941aa4][SLP] perl
78483 c406ec48 21802   550   550 0000000 [SLPQ select 0xc0941aa4][SLP] perl
73847 c43ac418 21802   551   551 0000000 [SLPQ select 0xc0941aa4][SLP] perl
73846 c42d4418 21802   550   550 0000000 [SLPQ select 0xc0941aa4][SLP] perl
68027 c43b2a3c 21802   550   550 0000000 [SLPQ select 0xc0941aa4][SLP] perl
67937 c3fcb000 21802   551   551 0000000 [SLPQ select 0xc0941aa4][SLP] perl
67936 c3fcb624 21802   550   550 0000000 [SLPQ select 0xc0941aa4][SLP] perl
67933 c42d4830 21802   551   551 0000000 [SLPQ select 0xc0941aa4][SLP] perl
67932 c38de624 21802   550   550 0000000 [SLPQ select 0xc0941aa4][SLP] perl
67904 c413b830 21802   551   551 0000000 [SLPQ select 0xc0941aa4][SLP] perl
67903 c3cbb624 21802   550   550 0000000 [SLPQ select 0xc0941aa4][SLP] perl
66287 c43afa3c	  0   650   650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
60279 c3f00830 21802   551   551 0000000 [SLPQ select 0xc0941aa4][SLP] perl
60278 c4244624 21802   551   551 0000000 [SLPQ filelist lock 0xc0938bd0][SLP] perl
60277 c3fcba3c 21802   550   550 0000000 [SLPQ select 0xc0941aa4][SLP] perl
60276 c3fca000 21802   550   550 0000000 [SLPQ select 0xc0941aa4][SLP] perl
60249 c406e418 21802   551   551 0000000 [SLPQ select 0xc0941aa4][SLP] perl
60248 c3f00c48 21802   550   550 0000000 [SLPQ select 0xc0941aa4][SLP] perl
51444 c3c3620c 21802   551   551 0000000 [SLPQ select 0xc0941aa4][SLP] perl
51443 c4066000 21802   550   550 0000000 [SLPQ select 0xc0941aa4][SLP] perl
51430 c3b99624 21802   551   551 0000000 [SLPQ select 0xc0941aa4][SLP] perl
51429 c41aa624 21802   550   550 0000000 [SLPQ select 0xc0941aa4][SLP] perl
51427 c40c4830 21802   551   551 0000000 [SLPQ select 0xc0941aa4][SLP] perl
51426 c3f0120c 21802   550   550 0000000 [SLPQ select 0xc0941aa4][SLP] perl
51231 c4375624 21802   551   551 0000000 [SLPQ filelist lock 0xc0938bd0][SLP] perl
51230 c3fcb830 21802   550   550 0000000 [SLPQ select 0xc0941aa4][SLP] perl
51228 c3d10a3c 21802   551   551 0000000 [SLPQ select 0xc0941aa4][SLP] perl
51227 c4140624 21802   550   550 0000000 [SLPQ select 0xc0941aa4][SLP] perl
51222 c413b20c 21802   551   551 0000000 [SLPQ filelist lock 0xc0938bd0][SLP] perl
51221 c38de418 21802   550   550 0000000 [SLPQ select 0xc0941aa4][SLP] perl
51218 c43b020c 21802   551   551 0000000 [SLPQ select 0xc0941aa4][SLP] perl
51217 c3cbac48 21802   550   550 0000000 [SLPQ filelist lock 0xc0938bd0][SLP] perl
51215 c43a2a3c 21802   551   551 0000000 [SLPQ select 0xc0941aa4][SLP] perl
51214 c437e20c 21802   551   551 0000000 [SLPQ select 0xc0941aa4][SLP] perl
51213 c4244418 21802   550   550 0000000 [SLPQ select 0xc0941aa4][SLP] perl
51212 c3f02624 21802   550   550 0000000 [SLPQ select 0xc0941aa4][SLP] perl
51211 c3f80a3c 21802   551   551 0000000 [SLPQ select 0xc0941aa4][SLP] perl
51206 c43b0624 21802   550   550 0000000 [SLPQ select 0xc0941aa4][SLP] perl
51161 c413520c 21802   551   551 0000000 [SLPQ select 0xc0941aa4][SLP] perl
51160 c42d5418 21802   550   550 0000000 [SLPQ select 0xc0941aa4][SLP] perl
51153 c40c0418 21802   551   551 0000000 [SLPQ filelist lock 0xc0938bd0][SLP] perl
50987 c3c36418	  0	1 50987 0000100 [SLPQ sbwait 0xc3da3c40][SLP] sendmail
36954 c4135000 21802   550   550 0000000 [SLPQ select 0xc0941aa4][SLP] perl
36941 c38dd418 21802   551   551 0000000 [SLPQ select 0xc0941aa4][SLP] perl
32803 c43af20c	  0 32793 32803 0004003 [SLPQ ttyin 0xc3871810][SLP][SWAP] zsh
32793 c43b2c48	127 32792 32793 0004002 [SLPQ pause 0xc43b2c7c][SLP][SWAP] zsh
32792 c42a7c48	127 32781 32781 0000100 [SLPQ select 0xc0941aa4][SLP][SWAP] sshd
32781 c3fcb418	  0   491 32781 0004100 [SLPQ sbwait 0xc4547e90][SLP][SWAP] sshd
31218 c3cbb000 21802   551   551 0000000 [SLPQ select 0xc0941aa4][SLP] perl
31217 c3f01418 21802   550   550 0000000 [SLPQ select 0xc0941aa4][SLP] perl
 5491 c43b0a3c	  0	1  5491 0000100 [SLPQ sbwait 0xc45473e8][SLP] sendmail
  795 c3d91c48	  0   778   795 0004002 [SLPQ wait 0xc3d91c48][SLP] sh
  778 c3d94418	  0   760   778 0004002 [SLPQ pause 0xc3d9444c][SLP][SWAP] zsh
  760 c3ea1000	  0	1   760 0004102 [SLPQ wait 0xc3ea1000][SLP][SWAP] login
  759 c3ea120c	  0	1   759 0004002 [SLPQ ttyin 0xc3737410][SLP][SWAP] getty
  758 c3ea1418	  0	1   758 0004002 [SLPQ ttyin 0xc3734c10][SLP][SWAP] getty
  757 c3ea1624	  0	1   757 0004002 [SLPQ ttyin 0xc3734410][SLP][SWAP] getty
  756 c3ea1830	  0	1   756 0004002 [SLPQ ttyin 0xc372f410][SLP][SWAP] getty
  755 c3ea1a3c	  0	1   755 0004002 [SLPQ ttyin 0xc3736410][SLP][SWAP] getty
  754 c3d18c48	  0	1   754 0004002 [SLPQ ttyin 0xc3730c10][SLP][SWAP] getty
  753 c3d18624	  0	1   753 0004002 [SLPQ ttyin 0xc3735810][SLP][SWAP] getty
  752 c38de000	  0	1   752 0004002 [SLPQ ttyin 0xc3734810][SLP][SWAP] getty
  719 c3d91418	  0	1   717 0000000 [SLPQ filelist lock 0xc0938bd0][SLP] snmpd
  709 c3d18a3c	  0	1   709 0000100 [SLPQ pause 0xc3d18a70][SLP][SWAP] sendmail
  706 c3d12418	  0	1   706 0000100 [SLPQ pause 0xc3d1244c][SLP][SWAP] sendmail
  703 c3d91000	  0	1   703 0000100 [SLPQ pause 0xc3d91034][SLP][SWAP] sendmail
  700 c3d91624	  0	1   700 0000100 [SLPQ pause 0xc3d91658][SLP][SWAP] sendmail
  696 c3d12a3c	  0	1   696 0000100 [SLPQ pause 0xc3d12a70][SLP][SWAP] sendmail
  692 c3d94000	  0	1   692 0000100 [SLPQ pause 0xc3d94034][SLP][SWAP] sendmail
  689 c3d94624	  0	1   689 0000100 [SLPQ pause 0xc3d94658][SLP][SWAP] sendmail
  686 c3d1020c	  0	1   686 0000100 [SLPQ pause 0xc3d10240][SLP] sendmail
  683 c3d10624	  0	1   683 0000100 [SLPQ pause 0xc3d10658][SLP][SWAP] sendmail
  680 c3d12000	  0	1   680 0000100 [SLPQ pause 0xc3d12034][SLP] sendmail
  678 c3d12624	  0	1   678 0000100 [SLPQ select 0xc0941aa4][SLP] sendmail
  664 c3d10000	  0	1   664 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
  650 c3cba000	  0	1   650 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
  636 c3d10418	  0	1   636 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sendmail
  620 c38d8418	  0   616   616 0000001 [SLPQ lockf 0xc41d3bc0][SLP][SWAP] saslauthd
  619 c3b99830	  0   616   616 0000001 [SLPQ lockf 0xc878a040][SLP][SWAP] saslauthd
  618 c3cba418	  0   616   616 0000001 [SLPQ lockf 0xc44dc900][SLP][SWAP] saslauthd
  617 c38d8624	  0   616   616 0000001 [SLPQ accept 0xc3cf0466][SLP][SWAP] saslauthd
  616 c3d10c48	  0	1   616 0000001 [SLPQ lockf 0xc44dc380][SLP][SWAP] saslauthd
  605 c3cba20c 21802	 1   605 0000000 [SLPQ select 0xc0941aa4][SLP] perl
  604 c3d12830	  0   577   604 0004003 [SLPQ ttyin 0xc3725810][SLP][SWAP] zsh
  577 c3c33830	127   576   577 0004002 [SLPQ pause 0xc3c33864][SLP][SWAP] zsh
  576 c3b9920c	127   574   574 0000100 [SLPQ select 0xc0941aa4][SLP][SWAP] sshd
  574 c38dd624	  0   491   574 0004100 [SLPQ sbwait 0xc3c30638][SLP][SWAP] sshd
  551 c38a5a3c 21802	 1   551 0000001 [SLPQ select 0xc0941aa4][SLP] perl
  550 c38ddc48 21802	 1   550 0000000 [SLPQ filelist lock 0xc0938bd0][SLP] perl
  546 c3b99a3c 21802	 1   546 0000001 [SLPQ select 0xc0941aa4][SLP] perl
  539 c38a5830 21802   534   534 0000000 [SLPQ select 0xc0941aa4][SLP][SWAP] httpd
  538 c38a8000 21802   534   534 0000000 [SLPQ select 0xc0941aa4][SLP][SWAP] httpd
  537 c38d820c 21802   534   534 0000000 [SLPQ select 0xc0941aa4][SLP][SWAP] httpd
  536 c3ba120c 21802   534   534 0000000 [SLPQ select 0xc0941aa4][SLP][SWAP] httpd
  535 c3ba1418 21802   534   534 0000000 [SLPQ select 0xc0941aa4][SLP][SWAP] httpd
  534 c38d8000 21802	 1   534 0000000 [SLPQ select 0xc0941aa4][SLP] httpd
  527 c38a5624	  0	1   527 0008180 (threaded)  perl
   thread 0xc41c1a80 ksegrp 0xc34db660 [SLPQ kserel 0xc34db694][SLP]
   thread 0xc879b300 ksegrp 0xc34db660 [SLPQ filelist lock 0xc0938bd0][SLP]
   thread 0xc43a6a80 ksegrp 0xc34db660 [SLPQ kserel 0xc34db694][SLP]
   thread 0xc42eec00 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc43a8480 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc59b5600 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc87e8480 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc4ade000 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc42ee480 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc87e5180 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc43a9300 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc87a2480 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc423a600 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc44f2600 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc5a0b780 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc5a36600 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc87e9780 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc42ee780 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc87e9300 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc87e7a80 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc5a20000 ksegrp 0xc34db660 [SLPQ filelist lock 0xc0938bd0][SLP]
   thread 0xc87e9c00 ksegrp 0xc34db660 [SLPQ filelist lock 0xc0938bd0][SLP]
   thread 0xc42efa80 ksegrp 0xc34db660 [SLPQ filelist lock 0xc0938bd0][SLP]
   thread 0xc41c0780 ksegrp 0xc34db660 [SLPQ filelist lock 0xc0938bd0][SLP]
   thread 0xc4099900 ksegrp 0xc34db660 [SLPQ filelist lock 0xc0938bd0][SLP]
   thread 0xc43c9780 ksegrp 0xc34db660 [SLPQ filelist lock 0xc0938bd0][SLP]
   thread 0xc43ca180 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc41c4180 ksegrp 0xc34db660 [SLPQ filelist lock 0xc0938bd0][SLP]
   thread 0xc41c1600 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc59e7c00 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc87a2a80 ksegrp 0xc34db660 [SLPQ filelist lock 0xc0938bd0][SLP]
   thread 0xc5a20d80 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc422d480 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc5a20600 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc3ea2900 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc41a1180 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc87a3780 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc4096480 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc59e7480 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc43a6900 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc442dc00 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc3f03000 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc5a20a80 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc4adf180 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc599d480 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc42ef600 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc442e600 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc4ade480 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc87e7180 ksegrp 0xc34db660 [SLPQ filelist lock 0xc0938bd0][SLP]
   thread 0xc879ba80 ksegrp 0xc34db660 [SLPQ filelist lock 0xc0938bd0][SLP]
   thread 0xc59b5180 ksegrp 0xc34db660 [SLPQ filelist lock 0xc0938bd0][SLP]
   thread 0xc4120d80 ksegrp 0xc34db660 [SLPQ filelist lock 0xc0938bd0][SLP]
   thread 0xc5a0b900 ksegrp 0xc34db660 [SLPQ filelist lock 0xc0938bd0][SLP]
   thread 0xc87e7600 ksegrp 0xc34db660 [SLPQ filelist lock 0xc0938bd0][SLP]
   thread 0xc599d900 ksegrp 0xc34db660 [SLPQ filelist lock 0xc0938bd0][SLP]
   thread 0xc423a900 ksegrp 0xc34db660 [SLPQ filelist lock 0xc0938bd0][SLP]
   thread 0xc3ea2600 ksegrp 0xc34db660 [SLPQ filelist lock 0xc0938bd0][SLP]
   thread 0xc87e8780 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc422d780 ksegrp 0xc34db660 [SLPQ filelist lock 0xc0938bd0][SLP]
   thread 0xc5a20900 ksegrp 0xc34db660 [SLPQ filelist lock 0xc0938bd0][SLP]
   thread 0xc5a36480 ksegrp 0xc34db660 [SLPQ filelist lock 0xc0938bd0][SLP]
   thread 0xc43ca300 ksegrp 0xc34db660 [SLPQ filelist lock 0xc0938bd0][SLP]
   thread 0xc41a1600 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc3c35780 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc3d96900 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc87ead80 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc4067000 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc3f28180 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc4ade600 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc8a96000 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc87e5900 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc422d180 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc87e9900 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc87e7480 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc44ef180 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc59b5900 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc87e5d80 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc599d180 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc59b5000 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc4ade780 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc87a4480 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc4121300 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc3f92900 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc422d600 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc4adf000 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc3f8fa80 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc43a8780 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc4adea80 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc3ea2480 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc423aa80 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc4adf780 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc87e5300 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc3f03180 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc4099300 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc442d480 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc4068780 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc4121d80 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc87a4780 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc4095d80 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc87a2000 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc87a4900 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc43c9d80 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc87e6d80 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc8a96600 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc43ca480 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc59b5d80 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc59e7300 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc3d17600 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc87e8c00 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc42f1600 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc43a6780 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc5987c00 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc422c600 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc442d300 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc8a97000 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc3ea3c00 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc43a8a80 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc44efd80 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc87ea780 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc5987000 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc4ade900 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc41c4300 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc5a20c00 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc87e6900 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc3d17480 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc4ae1c00 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc43cad80 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc87ea900 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc422dd80 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc87e7d80 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc87e9180 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc4adf900 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc4aeb780 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc5a20780 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc87a2180 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc59e7d80 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc43c9180 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc42f1480 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc5987600 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc41c1780 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc87a4d80 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc41c1180 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc44ef480 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc4ae1a80 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc5987480 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc87a3900 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc87e9480 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc44ef900 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc3d95d80 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc599d600 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc5987900 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc422cc00 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc599d000 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc3ea2300 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc87e6480 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc42f1d80 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc87e8600 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc4120780 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc599d780 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc41bfc00 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc4aeb900 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc42ee900 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc4067a80 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc43a8600 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc4121600 ksegrp 0xc34db660 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc38df900 ksegrp 0xc354d8a0 [SLPQ ksesigwait 0xc38a575c][SLP]
  520 c38dd830 21803	 1    69 000c083 (threaded)  perl
   thread 0xc43a9600 ksegrp 0xc354dcc0 [SLPQ kserel 0xc354dcf4][SLP]
   thread 0xc8a97a80 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc41a0c00 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc5a0bc00 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc87e9d80 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc43a8900 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc59e7000 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc3f8f180 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc8a97d80 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc599dc00 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc4ae1480 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc42ee300 ksegrp 0xc354dcc0 [SLPQ kserel 0xc354dcf4][SLP]
   thread 0xc87e8a80 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc87e5600 ksegrp 0xc354dcc0 [SLPQ filelist lock 0xc0938bd0][SLP]
   thread 0xc87e7300 ksegrp 0xc354dcc0 [SLPQ filelist lock 0xc0938bd0][SLP]
   thread 0xc5a0bd80 ksegrp 0xc354dcc0 [SLPQ filelist lock 0xc0938bd0][SLP]
   thread 0xc442da80 ksegrp 0xc354dcc0 [SLPQ filelist lock 0xc0938bd0][SLP]
   thread 0xc879b480 ksegrp 0xc354dcc0 [SLPQ filelist lock 0xc0938bd0][SLP]
   thread 0xc879b780 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc44f2180 ksegrp 0xc354dcc0 [SLPQ filelist lock 0xc0938bd0][SLP]
   thread 0xc3f90780 ksegrp 0xc354dcc0 [SLPQ filelist lock 0xc0938bd0][SLP]
   thread 0xc5987d80 ksegrp 0xc354dcc0 [SLPQ filelist lock 0xc0938bd0][SLP]
   thread 0xc879bc00 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc87e7c00 ksegrp 0xc354dcc0 [SLPQ filelist lock 0xc0938bd0][SLP]
   thread 0xc42ee180 ksegrp 0xc354dcc0 [SLPQ filelist lock 0xc0938bd0][SLP]
   thread 0xc87a2300 ksegrp 0xc354dcc0 [SLPQ filelist lock 0xc0938bd0][SLP]
   thread 0xc59b5480 ksegrp 0xc354dcc0 [SLPQ filelist lock 0xc0938bd0][SLP]
   thread 0xc442e900 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc5987180 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc5987780 ksegrp 0xc354dcc0 [SLPQ filelist lock 0xc0938bd0][SLP]
   thread 0xc5a0b480 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc599dd80 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc87ea180 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc3c34780 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc59b5c00 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc41c0900 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc43a9900 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc3f92c00 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc87a3480 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc87e9000 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc3f81d80 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc41a0d80 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc5a36000 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc43a9480 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc8a96a80 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc42f1900 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc87a2d80 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc41c0600 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc4120000 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc87e8900 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc87e7780 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc87a3300 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc87e9a80 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc423a480 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc4099d80 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc5a20480 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc442d180 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc59e7180 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc4096d80 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc43ca600 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc5a36180 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc423ad80 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc442ec00 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc41c4c00 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc87e5c00 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc4096a80 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc3ea3600 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc41c4000 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc87a4c00 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc43c9300 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc42eed80 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc43a9a80 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc4adfd80 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc4ade180 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc41a1480 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc87a4000 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc5987a80 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc87e6a80 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc43a9780 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc8a96180 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc4aeb000 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc42ef480 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc4120a80 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc4121180 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc44ef300 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc87a3a80 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc8a96780 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc59b5780 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc43a9180 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc87e8180 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc4adf600 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc423a300 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc41bfd80 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc3ea3480 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc422c780 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc422c300 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc422d900 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc44f2000 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc8a97600 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc59e7900 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc3f28780 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc4aeb180 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc87e8d80 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc43a8300 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc3f28c00 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc43c9600 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc43c9900 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc41c0a80 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc41bf600 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc422c900 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc43c9a80 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc42f1a80 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc3ea3900 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc43caa80 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc43ca900 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc43c9c00 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc442d600 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc8a96300 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc87ea480 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc43a8180 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc41c1c00 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc3b9a000 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc41bf180 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc4121c00 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc4aebc00 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc422da80 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc87e5000 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc422c180 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc87eaa80 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc42ef300 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc87e6000 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc42f1c00 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc59e7a80 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc41c0d80 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc4adfc00 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc442e780 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc4099000 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc87a4300 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc4121000 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc8a97180 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc4121480 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc59b5300 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc879b600 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc87e9600 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc3f90c00 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc8a96d80 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc8a97300 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc4122a80 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc42eea80 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc5987300 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc423a780 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc4068300 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc4ae1180 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc3f8f480 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc442e180 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc4ae1900 ksegrp 0xc354dcc0 [SLPQ select 0xc0941aa4][SLP]
   thread 0xc38e0180 ksegrp 0xc354d900 [SLPQ ksesigwait 0xc38dd968][SLP]
  519 c38a5c48	  0	1    69 0004002 [SLPQ sbwait 0xc39cabc8][SLP][SWAP] perl
  504 c38a5418	  0	1   504 0000000 [SLPQ nanslp 0xc093af0c][SLP] cron
  491 c38a5000	  0	1   491 0000100 [SLPQ filelist lock 0xc0938bd0][SLP] sshd
  474 c38dd20c	  0	1   474 0000000 [SLPQ select 0xc0941aa4][SLP] ntpd
  381 c38a520c	  0	1   381 0000000 [SLPQ select 0xc0941aa4][SLP] syslogd
  352 c38d8830	  0	1   352 0000000 [SLPQ select 0xc0941aa4][SLP][SWAP] devd
  268 c38dda3c	 64   262   262 0000100 [SLPQ bpf 0xc38dce00][SLP] pflogd
  262 c38d8a3c	  0	1   262 0000000 [SLPQ sbwait 0xc39ca370][SLP][SWAP] pflogd
   68 c38a8418	  0	0     0 0000204 [SLPQ - 0xef03ed08][SLP] schedcpu
   67 c38a8624	  0	0     0 0000204 [SLPQ - 0xc094a02c][SLP] nfsiod 3
   66 c38a8830	  0	0     0 0000204 [SLPQ - 0xc094a028][SLP] nfsiod 2
   65 c38a8a3c	  0	0     0 0000204 [SLPQ - 0xc094a024][SLP] nfsiod 1
   64 c38a8c48	  0	0     0 0000204 [SLPQ - 0xc094a020][SLP] nfsiod 0
   63 c355a20c	  0	0     0 0000204 [SLPQ vlruwt 0xc355a20c][SLP] vnlru
   62 c355a418	  0	0     0 0000204 [SLPQ syncer 0xc093ac7c][SLP] syncer
   61 c355a624	  0	0     0 0000204 [SLPQ psleep 0xc0941fec][SLP] bufdaemon
   60 c355a830	  0	0     0 000020c [SLPQ pgzero 0xc09505e4][SLP] pagezero
   59 c355aa3c	  0	0     0 0000204 [SLPQ psleep 0xc0950134][SLP] vmdaemon
   58 c355ac48	  0	0     0 0000204 [SLPQ psleep 0xc09500f0][SLP] pagedaemon
   57 c3603000	  0	0     0 0000204 [IWAIT] swi0: sio
   56 c360320c	  0	0     0 0000204 [SLPQ - 0xc372903c][SLP] fdc0
   55 c3603418	  0	0     0 0000204 [SLPQ aifthd 0xc3603418][SLP] aac0aif
   54 c3603624	  0	0     0 0000204 [SLPQ idle 0xc3528c00][SLP] aic_recovery0
   53 c3603830	  0	0     0 0000204 [SLPQ idle 0xc3528c00][SLP] aic_recovery0
   52 c3603a3c	  0	0     0 0000204 [IWAIT] swi6:+
   51 c3603c48	  0	0     0 0000204 [IWAIT] swi6: task queue
    9 c354b624	  0	0     0 0000204 [SLPQ - 0xc3563600][SLP] acpi_task2
    8 c354b830	  0	0     0 0000204 [SLPQ - 0xc3563600][SLP] acpi_task1
    7 c354ba3c	  0	0     0 0000204 [SLPQ - 0xc3563600][SLP] acpi_task0
   50 c354bc48	  0	0     0 0000204 [IWAIT] swi2: cambio
    6 c3559000	  0	0     0 0000204 [SLPQ - 0xc3563780][SLP] kqueue taskq
   49 c355920c	  0	0     0 0000204 [IWAIT] swi5:+
    5 c3559418	  0	0     0 0000204 [SLPQ - 0xc3563980][SLP] thread taskq
   48 c3559624	  0	0     0 0000204 [SLPQ - 0xc0935c60][SLP] yarrow
    4 c3559830	  0	0     0 0000204 [SLPQ - 0xc0938428][SLP] g_down
    3 c3559a3c	  0	0     0 0000204 [SLPQ - 0xc0938424][SLP] g_up
    2 c3559c48	  0	0     0 0000204 [SLPQ - 0xc093841c][SLP] g_event
   47 c355a000	  0	0     0 0000204 [IWAIT] swi1: net
   46 c353bc48	  0	0     0 0000204 [IWAIT] swi3: vm
   45 c354a000	  0	0     0 000020c [IWAIT] swi4: clock sio
   44 c354a20c	  0	0     0 0000204 [IWAIT] irq0:
   43 c354a418	  0	0     0 0000204 [IWAIT] irq31: aac0
   42 c354a624	  0	0     0 0000204 [IWAIT] irq30: ahc0
   41 c354a830	  0	0     0 0000204 [IWAIT] irq29:
   40 c354aa3c	  0	0     0 0000204 [IWAIT] irq28:
   39 c354ac48	  0	0     0 0000204 [IWAIT] irq27:
   38 c354b000	  0	0     0 0000204 [IWAIT] irq26:
   37 c354b20c	  0	0     0 0000204 [IWAIT] irq25:
   36 c354b418	  0	0     0 0000204 [IWAIT] irq24:
   35 c3529624	  0	0     0 0000204 [IWAIT] irq23:
   34 c3529830	  0	0     0 0000204 [IWAIT] irq22: fxp0
   33 c3529a3c	  0	0     0 0000204 [IWAIT] irq21:
   32 c3529c48	  0	0     0 0000204 [IWAIT] irq20:
   31 c353b000	  0	0     0 0000204 [IWAIT] irq19:
   30 c353b20c	  0	0     0 0000204 [IWAIT] irq18: fxp1
   29 c353b418	  0	0     0 0000204 [IWAIT] irq17: bge0
   28 c353b624	  0	0     0 0000204 [IWAIT] irq16: fxp2
   27 c353b830	  0	0     0 0000204 [IWAIT] irq15: ata1
   26 c353ba3c	  0	0     0 0000204 [IWAIT] irq14: ata0
   25 c34dc20c	  0	0     0 0000204 [IWAIT] irq13:
   24 c34dc418	  0	0     0 0000204 [IWAIT] irq12:
   23 c34dc624	  0	0     0 0000204 [IWAIT] irq11:
   22 c34dc830	  0	0     0 0000204 [IWAIT] irq10:
   21 c34dca3c	  0	0     0 0000204 [IWAIT] irq9: acpi0
   20 c34dcc48	  0	0     0 0000204 [IWAIT] irq8:
   19 c3529000	  0	0     0 0000204 [IWAIT] irq7: ppc0
   18 c352920c	  0	0     0 0000204 [IWAIT] irq6: fdc0
   17 c3529418	  0	0     0 0000204 [IWAIT] irq5:
   16 c34d7000	  0	0     0 0000204 [IWAIT] irq4: sio0
   15 c34d720c	  0	0     0 0000204 [IWAIT] irq3: sio1
   14 c34d7418	  0	0     0 0000204 [IWAIT] irq2:
   13 c34d7624	  0	0     0 0000204 [IWAIT] irq1: atkbd0
   12 c34d7830	  0	0     0 000020c [CPU 0] idle: cpu0
   11 c34d7a3c	  0	0     0 000020c [Can run] idle: cpu1
    1 c34d7c48	  0	0     1 0004200 [SLPQ wait 0xc34d7c48][SLP] init
   10 c34dc000	  0	0     0 0000204 [SLPQ ktrace 0xc0938e78][SLP] ktrace
    0 c0938520	  0	0     0 0000200 [IWAIT] swapper
db> panic
panic: from debugger
cpuid = 1
KDB: stack backtrace:
kdb_backtrace(100,c3c34c00,0,c089e89c,c089e8b8) at 0xc0659ab5 = kdb_backtrace+0x29
panic(c084a8b4,f1325a04,c0468c70,c093ace0,0) at 0xc06408b8 = panic+0x114
db_panic(c093ace0,0,ffffffff,f1325978,0) at 0xc0468cd9 = db_panic+0xd
db_command(c09217e4,c08a7820,c089e89c,c089e8b8,c084a8c2) at 0xc0468c70 = db_command+0x264
db_command_loop(0,0,f1325a30,f1325a1c,f1325a64) at 0xc0468d38 = db_command_loop+0x5c
db_trap(3,0,c3c34c00,3,c3c34c00) at 0xc046a949 = db_trap+0xdd
kdb_trap(3,0,f1325abc) at 0xc0659dd0 = kdb_trap+0x88
trap(f1320008,c0650028,c0870028,f9,c372ec00) at 0xc0817c88 = trap+0x498
calltrap() at 0xc0804cba = calltrap+0x5
--- trap 0x3, eip = 0xc0659b37, esp = 0xf1325afc, ebp = 0xf1325afc ---
kdb_enter(c0895a5e) at 0xc0659b37 = kdb_enter+0x2b
siointr1(c372ec00) at 0xc07f09b2 = siointr1+0xce
siointr(c372ec00) at 0xc07f0792 = siointr+0x5e
intr_execute_handlers(c3528090,f1325b68,4,f1325bcc,c0805073) at 0xc0808985 = intr_execute_handlers+0x85
lapic_handle_intr(34) at 0xc080b042 = lapic_handle_intr+0x2e
Xapic_isr1() at 0xc0805073 = Xapic_isr1+0x33
--- interrupt, eip = 0xc0685d59, esp = 0xf1325bac, ebp = 0xf1325bcc ---
unp_gc(c43be2c8) at 0xc0685d59 = unp_gc+0xf1
unp_detach(c43b7000,f1325c08,c067a188,c43be2c8,0) at 0xc06840fa = unp_detach+0xae
uipc_detach(c43be2c8) at 0xc0682f8e = uipc_detach+0x82
soclose(c43be2c8,c40fb6c0,0,f1325c34,c0620c10) at 0xc067a188 = soclose+0x278
soo_close(c40fb6c0,c3c34c00) at 0xc0669fb3 = soo_close+0x63
fdrop_locked(c40fb6c0,c3c34c00,c3a3d800,f1325cb4,c061f0fb) at 0xc0620c10 = fdrop_locked+0xac
fdrop(c40fb6c0,c3c34c00,c066371f,c40fb6c0,c878ac00) at 0xc0620b5d = fdrop+0x41
closef(c40fb6c0,c3c34c00,0,c3c34c00,6) at 0xc061f0fb = closef+0x427
close(c3c34c00,f1325d04,1,0,292) at 0xc061c4bd = close+0x211
syscall(3b,1003b,bfbf003b,80d58e7,80e8780) at 0xc0818557 = syscall+0x2bf
Xint0x80_syscall() at 0xc0804d0f = Xint0x80_syscall+0x1f
--- syscall (6, FreeBSD ELF32, close), eip = 0x282cb7d3, esp = 0xbfbfae4c, ebp = 0xbfbfae68 ---
KDB: stack backtrace:
kdb_backtrace(28d5a61e,53755e05,ffc03014,c3c34c00,c084a8b4) at 0xc0659ab5 = kdb_backtrace+0x29
mi_switch(1,0,f1325924,c0640190,c3c34c00) at 0xc0647162 = mi_switch+0xf6
sched_bind(c3c34c00,0) at 0xc065287c = sched_bind+0x44
boot(104,104,c3c34c00,0,c089e89c) at 0xc0640190 = boot+0x64
panic(c084a8b4,f1325a04,c0468c70,c093ace0,0) at 0xc064095d = panic+0x1b9
db_panic(c093ace0,0,ffffffff,f1325978,0) at 0xc0468cd9 = db_panic+0xd
db_command(c09217e4,c08a7820,c089e89c,c089e8b8,c084a8c2) at 0xc0468c70 = db_command+0x264
db_command_loop(0,0,f1325a30,f1325a1c,f1325a64) at 0xc0468d38 = db_command_loop+0x5c
db_trap(3,0,c3c34c00,3,c3c34c00) at 0xc046a949 = db_trap+0xdd
kdb_trap(3,0,f1325abc) at 0xc0659dd0 = kdb_trap+0x88
trap(f1320008,c0650028,c0870028,f9,c372ec00) at 0xc0817c88 = trap+0x498
calltrap() at 0xc0804cba = calltrap+0x5
--- trap 0x3, eip = 0xc0659b37, esp = 0xf1325afc, ebp = 0xf1325afc ---
kdb_enter(c0895a5e) at 0xc0659b37 = kdb_enter+0x2b
siointr1(c372ec00) at 0xc07f09b2 = siointr1+0xce
siointr(c372ec00) at 0xc07f0792 = siointr+0x5e
intr_execute_handlers(c3528090,f1325b68,4,f1325bcc,c0805073) at 0xc0808985 = intr_execute_handlers+0x85
lapic_handle_intr(34) at 0xc080b042 = lapic_handle_intr+0x2e
Xapic_isr1() at 0xc0805073 = Xapic_isr1+0x33
--- interrupt, eip = 0xc0685d59, esp = 0xf1325bac, ebp = 0xf1325bcc ---
unp_gc(c43be2c8) at 0xc0685d59 = unp_gc+0xf1
unp_detach(c43b7000,f1325c08,c067a188,c43be2c8,0) at 0xc06840fa = unp_detach+0xae
uipc_detach(c43be2c8) at 0xc0682f8e = uipc_detach+0x82
soclose(c43be2c8,c40fb6c0,0,f1325c34,c0620c10) at 0xc067a188 = soclose+0x278
soo_close(c40fb6c0,c3c34c00) at 0xc0669fb3 = soo_close+0x63
fdrop_locked(c40fb6c0,c3c34c00,c3a3d800,f1325cb4,c061f0fb) at 0xc0620c10 = fdrop_locked+0xac
fdrop(c40fb6c0,c3c34c00,c066371f,c40fb6c0,c878ac00) at 0xc0620b5d = fdrop+0x41
closef(c40fb6c0,c3c34c00,0,c3c34c00,6) at 0xc061f0fb = closef+0x427
close(c3c34c00,f1325d04,1,0,292) at 0xc061c4bd = close+0x211
syscall(3b,1003b,bfbf003b,80d58e7,80e8780) at 0xc0818557 = syscall+0x2bf
Xint0x80_syscall() at 0xc0804d0f = Xint0x80_syscall+0x1f
--- syscall (6, FreeBSD ELF32, close), eip = 0x282cb7d3, esp = 0xbfbfae4c, ebp = 0xbfbfae68 ---
[Previous attempts at using "call boot(0)"/etc. produced a solid hang that
required physical access to the Reset button / power switch.]
db> call cpu_reset()
cpu_reset: Restarting BSP
cpu_reset_proxy: Stopped CPU 1


[Seeing other posts that request kernel config additions to assist with
debugging, I then tried to install a kernel with the added config of:
options WITNESS
and booting failed as such:]

/boot/kernel/acpi.ko text=0x40c0c data=0x2160+0x1090 syms=[0x4+0x7810+0x4+0xa292]
KDB: debugger backends: ddb
KDB: current backend: ddb
Copyright (c) 1992-2005 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
	The Regents of the University of California. All rights reserved.
FreeBSD 6.0-RC1 #1: Wed Oct 19 09:45:23 CDT 2005
    root_at_[host-elided]:/usr/obj/usr/src/sys/SMP_DDB
WARNING: WITNESS option enabled, expect reduced performance.
Timecounter \"i8254\" frequency 1193182 Hz quality 0
CPU: Intel Pentium III (997.46-MHz 686-class CPU)
  Origin = \"GenuineIntel\"  Id = 0x68a  Stepping = 10
  Features=0x383fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE>
real memory  = 2147418112 (2047 MB)
avail memory = 2096447488 (1999 MB)
ACPI APIC Table: <DELL	 PE2550  >
FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
 cpu0 (BSP): APIC ID:  1
 cpu1 (AP): APIC ID:  0
ioapic0: Changing APIC ID to 2
ioapic1: Changing APIC ID to 3
MADT: Forcing active-low polarity and level trigger for SCI
ioapic0 <Version 1.1> irqs 0-15 on motherboard
ioapic1 <Version 1.1> irqs 16-31 on motherboard
panic: spin lock hptlock not in order list
cpuid = 0
KDB: stack backtrace:
kdb_backtrace(100,c0903380,0,c085e142,c08a28ac) at 0xc0646395 = kdb_backtrace+0x29
panic(c084621f,c085e142,c090c6c0,0,c0845c3a) at 0xc062fd4c = panic+0x114
enroll(c085e142,c08a28ac,c096d720,1,c0c20d5c) at 0xc064f61b = enroll+0xef
witness_init(c096d720,c34bfb74,c08dd510,c28000,c0c20d74) at 0xc064e6dd = witness_init+0x11d
mtx_init(c096d720,c085e142,0,1,c0c20d88) at 0xc0628d63 = mtx_init+0x8f
hpt_init(0,c1ec00,c1e000,0,c04457e5) at 0xc07ac352 = hpt_init+0x16
mi_startup() at 0xc060b1be = mi_startup+0x96
begin() at 0xc04457e5 = begin+0x2c
KDB: enter: panic
[thread pid 0 tid 0 ]
Stopped at	0xc0646417 = kdb_enter+0x2b:	nop
db>
db> where
Tracing pid 0 tid 0 td 0xc0903380
kdb_enter(c0843318) at 0xc0646417 = kdb_enter+0x2b
panic(c084621f,c085e142,c090c6c0,0,c0845c3a) at 0xc062fd5f = panic+0x127
enroll(c085e142,c08a28ac,c096d720,1,c0c20d5c) at 0xc064f61b = enroll+0xef
witness_init(c096d720,c34bfb74,c08dd510,c28000,c0c20d74) at 0xc064e6dd = witness_init+0x11d
mtx_init(c096d720,c085e142,0,1,c0c20d88) at 0xc0628d63 = mtx_init+0x8f
hpt_init(0,c1ec00,c1e000,0,c04457e5) at 0xc07ac352 = hpt_init+0x16
mi_startup() at 0xc060b1be = mi_startup+0x96
begin() at 0xc04457e5 = begin+0x2c
db> show pcpu
cpuid	     = 0
curthread    = 0xc0903380: pid 0 \"swapper\"
curpcb	     = 0xc0c20d90
fpcurthread  = none
idlethread   = 0xc34d8900: pid 12 \"idle: cpu0\"
APIC ID      = 1
currentldt   = 0x50
spin locks held:
db> show pcpu 0
cpuid	     = 0
curthread    = 0xc0903380: pid 0 \"swapper\"
curpcb	     = 0xc0c20d90
fpcurthread  = none
idlethread   = 0xc34d8900: pid 12 \"idle: cpu0\"
APIC ID      = 1
currentldt   = 0x50
spin locks held:
db> show pcpu 1
cpuid	     = 1
curthread    = none
curpcb	     = 0
fpcurthread  = none
idlethread   = 0xc34d8780: pid 11 \"idle: cpu1\"
APIC ID      = 0
currentldt   = 0x50
spin locks held:
db> show sysctllock
No such command
db> print sysctllock
c0905920
db> show reg
cs		  0x20
ds	    0xc0840028
es	    0xc0640028	blist_create+0x4
fs	    0xc0c20008
ss		  0x28
eax		  0x12
ecx	    0xc1033000
edx		     0
ebx	    0xc0c20d0c
esp	    0xc0c20ce0
ebp	    0xc0c20ce0
esi		   0x1
edi	    0xc084621f
eip	    0xc0646417	kdb_enter+0x2b
efl	       0x80292
dr0		     0
dr1		     0
dr2		     0
dr3		     0
dr4	    0xffff0ff0
dr5		 0x400
dr6	    0xffff0ff0
dr7		 0x400
0xc0646417 = kdb_enter+0x2b:	nop
db> show witness
Sleep locks:
0 module subsystem sx lock -- last acquired _at_ /usr/src/sys/kern/kern_module.c:115
0 kernel environment -- last acquired _at_ /usr/src/sys/kern/subr_hints.c:117
0 user map -- last acquired _at_ /usr/src/sys/vm/vm_map.c:2489
1  system map -- last acquired _at_ /usr/src/sys/vm/vm_kern.c:295
2   KMAP ENTRY -- last acquired _at_ /usr/src/sys/vm/uma_core.c:2631
3    UMA zone -- last acquired _at_ /usr/src/sys/vm/uma_core.c:1898
2   kmem object -- last acquired _at_ /usr/src/sys/vm/vm_kern.c:397
3    SYSMAPS -- last acquired _at_ /usr/src/sys/i386/i386/pmap.c:2410
3    vm page queue mutex -- last acquired _at_ /usr/src/sys/vm/vm_kern.c:404
4     pmap -- last acquired _at_ /usr/src/sys/i386/i386/pmap.c:1878
4     vnode interlock -- last acquired _at_ order list:0
5      cdev -- last acquired _at_ /usr/src/sys/kern/kern_conf.c:62
2   kernel object -- last acquired _at_ /usr/src/sys/vm/vm_object.c:454
3    SYSMAPS -- (already displayed)
3    vm page queue mutex -- (already displayed)
2   UMA lock -- last acquired _at_ /usr/src/sys/vm/uma_core.c:914
3    UMA zone -- (already displayed)
0 nfsd_mtx -- last acquired _at_ order list:0
3  so_snd -- last acquired _at_ order list:0
4   so_rcv -- last acquired _at_ order list:0
5    radix node head -- last acquired _at_ order list:0
6     rtentry -- last acquired _at_ order list:0
7      ifaddr -- last acquired _at_ order list:0
5    sellck -- last acquired _at_ order list:0
0 bpf global lock -- last acquired _at_ order list:0
1  bpf interface lock -- last acquired _at_ order list:0
2   bpf cdev lock -- last acquired _at_ order list:0
0 ddp_list_mtx -- last acquired _at_ order list:0
1  ddp_mtx -- last acquired _at_ order list:0
0 slip_mtx -- last acquired _at_ order list:0
1  slip sc_mtx -- last acquired _at_ order list:0
0 tcp -- last acquired _at_ order list:0
1  tcpinp -- last acquired _at_ order list:0
3   so_snd -- (already displayed)
0 udp -- last acquired _at_ order list:0
1  udpinp -- last acquired _at_ order list:0
3   so_snd -- (already displayed)
0 unp -- last acquired _at_ order list:0
3  so_snd -- (already displayed)
0 in_multi_mtx -- last acquired _at_ order list:0
1  igmp_mtx -- last acquired _at_ order list:0
2   if_addr_mtx -- last acquired _at_ order list:0
0 Giant -- last acquired _at_ /usr/src/sys/kern/subr_witness.c:513
1  eventhandler list -- last acquired _at_ /usr/src/sys/kern/subr_eventhandler.c:132
1  vm object_list -- last acquired _at_ /usr/src/sys/vm/vm_object.c:229
1  system map -- (already displayed)
1  kobj -- last acquired _at_ /usr/src/sys/kern/subr_kobj.c:148
1  eventhandler -- last acquired _at_ /usr/src/sys/kern/subr_eventhandler.c:82
1  filedesc structure -- last acquired _at_ /usr/src/sys/kern/kern_descrip.c:1512
2   accept -- last acquired _at_ order list:0
3    so_snd -- (already displayed)
2   pipe mutex -- last acquired _at_ order list:0
3    sigio lock -- last acquired _at_ order list:0
4     process group -- last acquired _at_ /usr/src/sys/kern/kern_fork.c:562
5      process lock -- last acquired _at_ /usr/src/sys/kern/kern_kthread.c:96
6	ktrace -- last acquired _at_ /usr/src/sys/kern/kern_fork.c:590
6	sigacts -- last acquired _at_ /usr/src/sys/kern/kern_kthread.c:98
6	session -- last acquired _at_ /usr/src/sys/kern/kern_fork.c:572
7	 uidinfo hash -- last acquired _at_ /usr/src/sys/kern/kern_resource.c:1032
8	  sleep mtxpool -- last acquired _at_ /usr/src/sys/kern/kern_resource.c:914
8	  uidinfo struct -- last acquired _at_ order list:0
9	   allprison -- last acquired _at_ order list:0
1  ithread -- last acquired _at_ /usr/src/sys/kern/kern_intr.c:275
1  malloc -- last acquired _at_ /usr/src/sys/kern/kern_malloc.c:598
1  kernel linker -- last acquired _at_ /usr/src/sys/kern/kern_linker.c:470
1  ACPI semaphore -- last acquired _at_ /usr/src/sys/modules/acpi/acpi/../../../dev/acpica/Osd/OsdSynch.c:301
1  TID lock -- last acquired _at_ /usr/src/sys/kern/subr_unit.c:592
1  standard object -- last acquired _at_ /usr/src/sys/vm/vm_glue.c:290
1  ttylist -- last acquired _at_ /usr/src/sys/kern/tty.c:2865
1  GEOM orphanage -- last acquired _at_ /usr/src/sys/geom/geom_event.c:300
0 proctree -- last acquired _at_ /usr/src/sys/kern/kern_fork.c:561
4  process group -- (already displayed)
1  allproc -- last acquired _at_ /usr/src/sys/kern/kern_fork.c:286
5   process lock -- (already displayed)
Spin locks:
0 ap boot -- last acquired _at_ order list:0
1  rm.mutex_mtx -- last acquired _at_ order list:0
2   sio -- last acquired _at_ order list:0
3    cy -- last acquired _at_ order list:0
4     uart_hwmtx -- last acquired _at_ order list:0
5      sabtty -- last acquired _at_ order list:0
6	zstty -- last acquired _at_ order list:0
7	 ng_node -- last acquired _at_ order list:0
8	  ng_worklist -- last acquired _at_ order list:0
9	   taskqueue_fast -- last acquired _at_ order list:0
10	    intr table -- last acquired _at_ /usr/src/sys/i386/i386/intr_machdep.c:89
11	     ithread table lock -- last acquired _at_ order list:0
12	      sleepq chain -- last acquired _at_ /usr/src/sys/kern/subr_sleepqueue.c:218
13	       sched lock -- last acquired _at_ /usr/src/sys/kern/kern_kthread.c:116
14		turnstile chain -- last acquired _at_ order list:0
15		 td_contested -- last acquired _at_ order list:0
16		  callout -- last acquired _at_ /usr/src/sys/kern/kern_timeout.c:439
17		   entropy harvest mutex -- last acquired _at_ order list:0
18		    allpmaps -- last acquired _at_ /usr/src/sys/i386/i386/pmap.c:1429
19		     vm page queue free mutex -- last acquired _at_ /usr/src/sys/vm/vm_page.c:787
20		      icu -- last acquired _at_ order list:0
21		       smp rendezvous -- last acquired _at_ order list:0
22			tlb -- last acquired _at_ order list:0
23			 clk -- last acquired _at_ /usr/src/sys/i386/isa/clock.c:260
24			  mutex profiling lock -- last acquired _at_ order list:0
25			   kse zombie lock -- last acquired _at_ order list:0
26			    ALD Queue -- last acquired _at_ order list:0
27			     pcicfg -- last acquired _at_ order list:0
28			      NDIS thread lock -- last acquired _at_ order list:0
29			       tw_osl_io_lock -- last acquired _at_ order list:0
30				tw_osl_q_lock -- last acquired _at_ order list:0
31				 tw_cl_io_lock -- last acquired _at_ order list:0
32				  tw_cl_intr_lock -- last acquired _at_ order list:0
33				   tw_cl_gen_lock -- last acquired _at_ order list:0
Locks which were never acquired:
LED sx
LED mtx
tty
nfslock
random reseed
devd
bio queue
GEOM topology
if_cloners lock
ifnet
acpica subsystem lock
ACPI global lock
sf_buf
taskqueue list
uma object
p_peers
struct pargs.ref
pbuf mutex
bdone lock
buffer daemon lock
needsbuffer lock
runningbufspace lock
buf queue lock
callout_wait_lock
ACPI embedded controller
ACPI CPU
ACPI PCI power methods
ACPI cmbat
ACPI generic battery
ACPI AC adapter
ACPI thermal zone
ACPI PCI link
ACPI root bus
ACPI PCI bus methods
ACPI lid
ACPI power resources
umtxq_lock
bounce pages lock
arc4_mtx
securelevel mutex lock
accounting
knlist lock for lockless objects
net80211 instances
rtsock route_cb lock
Softdep Lock
domain list
rawcb
pfil_head_list lock
mountlist
pmc shared lock
accept_filter_mtx
protect sysfilt_ops
kqueue order
UUID generator mutex lock
unit# allocation
Name Cache
fifo mutex
so_glabel
encapmtx
DEVFS ruleset lock
fdesc
filelist lock
sysctl lock
phys_pager list
dev_pager list
dev_pager create
swapdev
swap_pager list
vm map sleep mutex
PMAP2
vm86 lock
db> cont
Uptime: 1s
Automatic reboot in 15 seconds - press a key on the console to abort
Rebooting...


And, attempts at adding:
options INVARIANTS

failed to compile with the following:

/usr/src/sys/dev/ata/atapi-cd.c: In function `acd_geom_attach':
/usr/src/sys/dev/ata/atapi-cd.c:181: warning: implicit declaration of function `_sx_assert'
/usr/src/sys/dev/ata/atapi-cd.c:181: warning: nested extern declaration of `_sx_assert'



Since I am having a hang that feels the same to me in both RELENG_5 and
RELENG_6, that's the main thing I would like to look at.  I am not very
conversant at what I will need for kernel debugging (knowing which
addresses to dump or structures to inspect) to debug this myself, but
if you can tell me any further commands to issue in DDB or which addresses
would assist in diagnosing or config options to add, that will be no
problem and I will be more than happy to assist in any way!


Thank you,
Philip
Received on Wed Oct 19 2005 - 14:23:31 UTC

This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:38:45 UTC