sk Panic in 8.0-CURRENT

Has anyone seen the following mutex panic in sk_jfree? The last time this 
system booted was Jan 31.


OK include /boot/testbed/current
/

testbed/current (8.0-CURRENT) loader file selected
-
unload complete
currdev set to disk1s1d:

/boot/kernel/kernel text=0x66d6e0 data=0xb3a00+0xaa07c 
syms=[0x4+0x76dc0+0x4+0x98529]
/boot/kernel/smbus.ko text=0xeec data=0x1a8+0x4 syms=[0x4+0x520+0x4+0x50a]
/boot/kernel/smb.ko text=0x1634 data=0x1b4+0x4 syms=[0x4+0x530+0x4+0x4df]
/boot/kernel/intpm.ko text=0x270c data=0x1e4+0x4 syms=[0x4+0x710+0x4+0x753]
/boot/kernel/if_sk.ko text=0x959c data=0x34c+0x8 syms=[0x4+0xd50+0x4+0xd1f]
/boot/kernel/if_vr.ko text=0x4560 data=0x234+0x4 syms=[0x4+0x930+0x4+0x8d1]
/boot/kernel/geom_concat.ko text=0x4178 data=0x204+0x8 
syms=[0x4+0x7b0+0x4+0x730]

new kernel has been loaded

OK boot
/boot/kernel/acpi.ko text=0x4d244 data=0x23a0+0x186c 
syms=[0x4+0x8660+0x4+0xb0e7]
GDB: no debug ports present
KDB: debugger backends: ddb
KDB: current backend: ddb
Copyright (c) 1992-2008 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
	The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 8.0-CURRENT #0: Mon Feb 11 12:35:05 PST 2008
    root_at_bob:/dsk02/obj/dsk03/src/cvs-current/src/sys/GENERIC
WARNING: WITNESS option enabled, expect reduced performance.
module_register: module sk/miibus already exists!
Module sk/miibus failed to register: 17
module_register: module skc/sk already exists!
Module skc/sk failed to register: 17
module_register: module pci/skc already exists!
Module pci/skc failed to register: 17
module_register: module vr/miibus already exists!
Module vr/miibus failed to register: 17
module_register: module pci/vr already exists!
Module pci/vr failed to register: 17
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: AMD Sempron(tm) Processor 3000+ (1599.99-MHz 686-class CPU)
  Origin = "AuthenticAMD"  Id = 0x40ff2  Stepping = 2
  Features=0x78bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MC
A,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2>
  Features2=0x2001<SSE3,CX16>
  AMD Features=0xea500800<SYSCALL,NX,MMX+,FFXSR,RDTSCP,LM,3DNow!+,3DNow!>
  AMD Features2=0x19<LAHF,ExtAPIC,CR8>
real memory  = 1542389760 (1470 MB)
avail memory = 1499729920 (1430 MB)
ACPI APIC Table: <K8M890 AWRDACPI>
ioapic0 <Version 0.3> irqs 0-23 on motherboard
ioapic1 <Version 0.3> irqs 24-47 on motherboard
kbd1 at kbdmux0
ath_hal: 0.9.20.3 (AR5210, AR5211, AR5212, RF5111, RF5112, RF2413, RF5413)
acpi0: <K8M890 AWRDACPI> on motherboard
acpi0: [ITHREAD]
acpi0: Power Button (fixed)
acpi0: reservation of 0, a0000 (3) failed
acpi0: reservation of 100000, 5bdf0000 (3) failed
Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
cpu0: <ACPI CPU> on acpi0
acpi_button0: <Power Button> on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <PCI-PCI bridge> at device 1.0 on pci0
pci1: <PCI bus> on pcib1
vgapci0: <VGA-compatible display> mem 0xc0000000-0xcfffffff,0xdd000000-0xddf
fffff irq 16 at device 0.0 on pci1
pcib2: <ACPI PCI-PCI bridge> irq 27 at device 2.0 on pci0
pci2: <ACPI PCI bus> on pcib2
pcib3: <ACPI PCI-PCI bridge> irq 31 at device 3.0 on pci0
pci3: <ACPI PCI bus> on pcib3
atapci0: <VIA 8237A SATA150 controller> port 0xff00-0xff07,0xfe00-0xfe03,0xf
d00-0xfd07,0xfc00-0xfc03,0xfb00-0xfb0f,0xf400-0xf4ff irq 21 at device 15.0 
on pci0
atapci0: [ITHREAD]
ata2: <ATA channel 0> on atapci0
ata2: [ITHREAD]
ata3: <ATA channel 1> on atapci0
ata3: [ITHREAD]
atapci1: <VIA 8237A UDMA133 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0
x376,0xfa00-0xfa0f at device 15.1 on pci0
ata0: <ATA channel 0> on atapci1
ata0: [ITHREAD]
ata1: <ATA channel 1> on atapci1
ata1: [ITHREAD]
uhci0: <VIA 83C572 USB controller> port 0xf900-0xf91f irq 20 at device 16.0 
on pci0
uhci0: [GIANT-LOCKED]
uhci0: [ITHREAD]
usb0: <VIA 83C572 USB controller> on uhci0
usb0: USB revision 1.0
uhub0: <VIA UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb0
uhub0: 2 ports with 2 removable, self powered
uhci1: <VIA 83C572 USB controller> port 0xf800-0xf81f irq 22 at device 16.1 
on pci0
uhci1: [GIANT-LOCKED]
uhci1: [ITHREAD]
usb1: <VIA 83C572 USB controller> on uhci1
usb1: USB revision 1.0
uhub1: <VIA UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb1
uhub1: 2 ports with 2 removable, self powered
uhci2: <VIA 83C572 USB controller> port 0xf700-0xf71f irq 21 at device 16.2 
on pci0
uhci2: [GIANT-LOCKED]
uhci2: [ITHREAD]
usb2: <VIA 83C572 USB controller> on uhci2
usb2: USB revision 1.0
uhub2: <VIA UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb2
uhub2: 2 ports with 2 removable, self powered
uhci3: <VIA 83C572 USB controller> port 0xf600-0xf61f irq 23 at device 16.3 
on pci0
uhci3: [GIANT-LOCKED]
uhci3: [ITHREAD]
usb3: <VIA 83C572 USB controller> on uhci3
usb3: USB revision 1.0
uhub3: <VIA UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb3
uhub3: 2 ports with 2 removable, self powered
ehci0: <VIA VT6202 USB 2.0 controller> mem 0xdffff000-0xdffff0ff irq 21 at 
device 16.4 on pci0
ehci0: [GIANT-LOCKED]
ehci0: [ITHREAD]
usb4: EHCI version 1.0
usb4: companion controllers, 2 ports each: usb0 usb1 usb2 usb3
usb4: <VIA VT6202 USB 2.0 controller> on ehci0
usb4: USB revision 2.0
uhub4: <VIA EHCI root hub, class 9/0, rev 2.00/1.00, addr 1> on usb4
uhub4: 8 ports with 8 removable, self powered
isab0: <PCI-ISA bridge> at device 17.0 on pci0
isa0: <ISA bus> on isab0
vr0: <VIA VT6102 Rhine II 10/100BaseTX> port 0xf200-0xf2ff mem 
0xdfffe000-0xdfffe0ff irq 23 at device 18.0 on pci0
vr0: Quirks: 0x0
miibus0: <MII bus> on vr0
rlphy0: <RTL8201L 10/100 media interface> PHY 1 on miibus0
rlphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
vr0: using obsoleted if_watchdog interface
vr0: Ethernet address: 00:16:17:8e:65:fe
vr0: [ITHREAD]
pcib4: <ACPI PCI-PCI bridge> at device 19.0 on pci0
pci4: <ACPI PCI bus> on pcib4
pci4: <multimedia> at device 1.0 (no driver attached)
pcib5: <ACPI PCI-PCI bridge> at device 19.1 on pci0
pci5: <ACPI PCI bus> on pcib5
ahc0: <Adaptec 2940 Ultra SCSI adapter> port 0xae00-0xaeff mem 
0xdf8ff000-0xdf8fffff irq 17 at device 4.0 on pci5
ahc0: [ITHREAD]
aic7880: Ultra Single Channel A, SCSI Id=7, 16/253 SCBs
skc0: <D-Link DGE-530T Gigabit Ethernet> port 0xac00-0xacff mem 
0xdf8f8000-0xdf8fbfff irq 18 at device 5.0 on pci5
skc0: DGE-530T Gigabit Ethernet Adapter rev. (0x1)
sk0: <Marvell Semiconductor, Inc. Yukon> on skc0
sk0: Ethernet address: 00:13:46:73:0c:0d
miibus1: <MII bus> on sk0
e1000phy0: <Marvell 88E1011 Gigabit PHY> PHY 0 on miibus1
e1000phy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseTX-FDX, 
auto
skc0: [ITHREAD]
fdc0: <floppy drive controller> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0
fdc0: [FILTER]
fd0: <1440-KB 3.5" drive> on fdc0 drive 0
sio0: configured irq 4 not in bitmap of probed irqs 0
sio0: port may not be enabled
sio0: configured irq 4 not in bitmap of probed irqs 0
sio0: port may not be enabled
sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 flags 0x10 on 
acpi0
sio0: type 16550A, console
sio0: [FILTER]
sio1: <16550A-compatible COM port> port 0x2f8-0x2ff irq 3 on acpi0
sio1: type 16550A
sio1: [FILTER]
pmtimer0 on isa0
orm0: <ISA Option ROM> at iomem 0xd4000-0xd87ff pnpid ORM0000 on isa0
atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
atkbd0: [ITHREAD]
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
Timecounter "TSC" frequency 1599991094 Hz quality 800
Timecounters tick every 1.000 msec
ad0: 76351MB <SAMSUNG SP0802N TK100-24> at ata0-master UDMA100
acd0: DVDR <HL-DT-ST DVDRAM GSA-H10N/JL10> at ata0-slave UDMA33
ad2: 117246MB <Maxtor 6Y120P0 YAR41BW0> at ata1-master UDMA133
ad3: 238475MB <WDC WD2500JB-00REA0 20.00K20> at ata1-slave UDMA100
Waiting 5 seconds for SCSI devices to settle
GEOM_LABEL: Label for provider acd0 is iso9660/CDROM.
da0 at ahc0 bus 0 target 1 lun 0
da0: <SEAGATE ST34520N 1487> Fixed Direct Access SCSI-2 device 
da0: 20.000MB/s transfers (20.000MHz, offset 15)
da0: Command Queueing Enabled
da0: 4340MB (8888924 512 byte sectors: 255H 63S/T 553C)
WARNING: WITNESS option enabled, expect reduced performance.
lock order reversal:
 1st 0xc3529e28 devfs (devfs) _at_ /dsk03/src/cvs-current/src/sys/kern/vfs_subr
.c:2061
 2nd 0xc343c6d4 devfsmount (devfsmount) _at_ /dsk03/src/cvs-current/src/sys/fs/
devfs/devfs_vnops.c:201
KDB: stack backtrace:
db_trace_self_wrapper(c0a391b6) at db_trace_self_wrapper+0x25
kdb_backtrace(c0b66f18,2,c3291d54,c0ba59a8,0,...) at kdb_backtrace+0x24
witness_checkorder(c343c6d4,9,c0a2c52d,c9,0,...) at witness_checkorder+0x58e
_sx_xlock(c343c6d4,0,c0a2c52d,c9,c3529dd0,2002,c3291cc0,c343c6d4,c0a2c52d,c7
) at _sx_xlock+0x52
devfs_allocv(c35f4b00,c3622000,e1c0dc68,c3291cc0,c343c6d4,0,c0a2c491,9d) at 
devfs_allocv+0xf1
devfs_root(c3622000,2,c0ba60d8,c3291cc0) at devfs_root+0x36
set_rootvnode(c0ba60c0,0,c0a416ef,5ed,25e,...) at set_rootvnode+0x1d
vfs_mountroot(c0b5c2c0,c3291cc0,c328fab0,c3291cc0,e1c0dcec,...) at 
vfs_mountroot+0x288
start_init(0,e1c0dd38) at start_init+0x4c
fork_exit(c06d9840,0,e1c0dd38) at fork_exit+0xb0
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe1c0dd70, ebp = 0 ---
Trying to mount root from ufs:/dev/ad0s1d
lock order reversal:
 1st 0xc35299e8 ufs (ufs) _at_ /dsk03/src/cvs-current/src/sys/kern/vfs_subr.c:2
061
 2nd 0xc3622000 vfslock (vfslock) _at_ /dsk03/src/cvs-current/src/sys/kern/vfs_
subr.c:364
KDB: stack backtrace:
db_trace_self_wrapper(c0a391b6) at db_trace_self_wrapper+0x25
kdb_backtrace(c0b66f40,2,c3291d54,c0ba5938,0,...) at kdb_backtrace+0x24
witness_checkorder(c3622000,1,c0a41d9c,16c) at witness_checkorder+0x58e
_lockmgr(c3622000,2001,c3622030,c0a41d9c,16c) at _lockmgr+0x124
vfs_busy(c3622000,0,0,c3291cc0) at vfs_busy+0x12f
lookup(e1c0dba8) at lookup+0x71d
namei(e1c0dba8) at namei+0x2b6
kern_unlink(c3291cc0,c0a41b3e,1) at kern_unlink+0x3e
vfs_mountroot_try(25e,c0a30cb8,0,0,c3291cc0,...) at vfs_mountroot_try+0x37a
vfs_mountroot(c0b5c2c0,c3291cc0,c328fab0,c3291cc0,e1c0dcec,...) at 
vfs_mountroot+0x338
start_init(0,e1c0dd38) at start_init+0x4c
fork_exit(c06d9840,0,e1c0dd38) at fork_exit+0xb0
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe1c0dd70, ebp = 0 ---
lock order reversal:
 1st 0xc3295044 user map (user map) _at_ /dsk03/src/cvs-current/src/sys/vm/vm_m
ap.c:3111
 2nd 0xc35297c8 ufs (ufs) _at_ /dsk03/src/cvs-current/src/sys/kern/vfs_subr.c:2
061
KDB: stack backtrace:
db_trace_self_wrapper(c0a391b6) at db_trace_self_wrapper+0x25
kdb_backtrace(c0b66ea0,0,c3291d54,c0ba59a8,0,...) at kdb_backtrace+0x24
witness_checkorder(c35297c8,1,c0a41d9c,80d) at witness_checkorder+0x58e
_lockmgr(c35297c8,3041,c35297f8,c0a41d9c,80d) at _lockmgr+0x124
ffs_lock(e1c0da90) at ffs_lock+0x6e
VOP_LOCK1_APV(c0b0a020,e1c0da90,c35297f8,c0b21a20,c3529770,...) at 
VOP_LOCK1_APV+0x83
_vn_lock(c3529770,3041,c0a41d9c,80d,c3529770,...) at _vn_lock+0xc9
vget(c3529770,3041,c3291cc0) at vget+0xd1
vnode_pager_lock(c1460380,c1460380,c1460380,0,c0a58d67,...) at 
vnode_pager_lock+0x13e
vm_fault(c3295000,80ce000,2,8,80ce1a0,...) at vm_fault+0x189
trap_pfault(c328fab0,0,c06fb711,6,c,...) at trap_pfault+0xf4
trap(e1c0dd38) at trap+0x24a
calltrap() at calltrap+0x6
--- trap 0xc, eip = 0x80480e2, esp = 0xbfbfef00, ebp = 0xbfbfef10 ---
Loading configuration files.
kernel dumps on /dev/ad0s1b
Entropy harvesting: interrupts ethernet point_to_point kickstart.
swapon: adding /dev/ad0s1b as swap device
swapon: /dev/ad2s1b: No such file or directory
swapon: adding /dev/ad3s1b as swap device
swapon: adding /dev/ad2p1 as swap device
Starting file system checks:
/dev/ad0s1d: FILE SYSTEM CLEAN; SKIPPING CHECKS
/dev/ad0s1d: clean, 2976187 free (14963 frags, 370153 blocks, 0.3% 
fragmentation)
/dev/ad3s1g: FILE SYSTEM CLEAN; SKIPPING CHECKS
/dev/ad3s1g: clean, 67600754 free (343074 frags, 8407210 blocks, 0.3% 
fragmentation)
/dev/ad0s1a: FILE SYSTEM CLEAN; SKIPPING CHECKS
/dev/ad0s1a: clean, 1975712 free (17584 frags, 244766 blocks, 0.4% 
fragmentation)
/dev/ad0s1e: FILE SYSTEM CLEAN; SKIPPING CHECKS
/dev/ad0s1e: clean, 4126258 free (10786 frags, 514434 blocks, 0.2% 
fragmentation)
/dev/da0p1: FILE SYSTEM CLEAN; SKIPPING CHECKS
/dev/da0p1: clean, 2249366 free (6782 frags, 280323 blocks, 0.2% 
fragmentation)
/dev/ad0s1f: FILE SYSTEM CLEAN; SKIPPING CHECKS
/dev/ad0s1f: clean, 2269772 free (15612 frags, 281770 blocks, 0.4% 
fragmentation)
/dev/ad0s2a: FILE SYSTEM CLEAN; SKIPPING CHECKS
/dev/ad0s2a: clean, 3969749 free (14645 frags, 494388 blocks, 0.3% 
fragmentation)
/dev/ad0s2d: FILE SYSTEM CLEAN; SKIPPING CHECKS
/dev/ad0s2d: clean, 3453823 free (11207 frags, 430327 blocks, 0.3% 
fragmentation)
/dev/ad0s2e: FILE SYSTEM CLEAN; SKIPPING CHECKS
/dev/ad0s2e: clean, 3428465 free (10849 frags, 427202 blocks, 0.2% 
fragmentation)
Setting hostuuid: ef98fa17-f448-11db-bd62-0016178e65fe.
Setting hostid: 0xf46bca72.
Mounting local file systems:mount: /amd64-6: No such file or directory
.
Setting hostname: bob.
net.inet6.ip6.auto_linklocal: 1 -> 0
vfs.nfsrv.async: 0 -> 1
net.inet.tcp.sendspace: 32768 -> 65536
kern.ipc.nmbclussters: k2304 -> 32768
0: link state changed to DOWN
vr0: link state changed to UP
sk0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
	options=b<RXCSUM,TXCSUM,VLAN_MTU>
	ether 00:13:46:73:0c:0d
	inet 10.1.1.7 netmask 0xffffff00 broadcast 10.1.1.255
	inet 10.1.1.2 netmask 0xffffffff broadcast 10.1.1.2
	media: Ethernet autoselect (none)
	status: no carrier
vr0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=8<VLAN_MTU>
	ether 00:16:17:8e:65:fe
	inet 10.1.2.7 netmask 0xffffff00 broadcast 10.1.2.255
	inet 10.1.2.2 netmask 0xffffffff broadcast 10.1.2.2
	media: Ethernet autoselect (100baseTX <full-duplex>)
	status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3 
	inet6 ::1 prefixlen 128 
	inet 127.0.0.1 netmask 0xff000000 
add net default: gateway 10.1.2.254
Additional routing options:.
Starting devd.
hw.acpi.cpu.cx_lowest: C1
sysctl: hw.acpi.cpu.cx_lowest: Invalid argument
Additional IP options:.
Mounting NFS file systems:sk0: link state changed to UP
panic: mtx_lock() of spin mutex (null) _at_ /dsk03/src/cvs-current/src/sys/modu
les/sk/../../dev/sk/if_sk.c:2439
cpuid = 0
KDB: enter: panic
[thread pid 12 tid 100038 ]
Stopped at      kdb_enter+0x34: movl    $0,kdb_why
db> bt
Tracing pid 12 tid 100038 td 0xc3363cc0
kdb_enter(c0a36183,c0a36183) at kdb_enter+0x34
panic(c0a34f9b,0,c0cefb36,987,e2583cc0,...) at panic+0x111
_mtx_lock_flags(e2586bbc,0,c0cefb36,987,c35d1000,...) at 
_mtx_lock_flags+0x70
sk_jfree(c341f000,e2583cc0) at sk_jfree+0x3a
mb_free_ext(c35d1000) at mb_free_ext+0x18f
m_freem(c35d1000) at m_freem+0x1f
arpintr(c35d1000) at arpintr+0xc0b
netisr_dispatch(12,c35d1000) at netisr_dispatch+0x5d
ether_demux(c33d5400,c35d1000) at ether_demux+0x1c9
ether_input(c33d5400,c35d1000,c33f36e0,0,c0cefb36,...) at ether_input+0x2f9
sk_jumbo_rxeof(c33f36e0,c341f000,c33d5400,0,c342d340,...) at 
sk_jumbo_rxeof+0x215
sk_intr(c33f3680) at sk_intr+0xac
ithread_loop(c342ab40,e2589d38) at ithread_loop+0x175
fork_exit(c06eded0,c342ab40,e2589d38) at fork_exit+0xb0
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe2589d70, ebp = 0 ---
db> 



-- 
Cheers,
Cy Schubert
<Cy.Schubert_at_komquats.com>