Two LOR in r261885

  

I tried searching the list and google and could not find a
reference which seemed to apply. Neither LOR dropped to the debugger.
This happens when under heavy load poudreire 16 processes. If its been
reported or nothing to worry about sorry for the noise. 

No sysctl,
make, loader changes - generic kernel. 

--mikej 

lock order reversal:

1st 0xfffffe0f9446ff28 bufwait (bufwait) _at_
/usr/src/sys/kern/vfs_bio.c:3081
 2nd 0xfffff801c946a400 dirhash
(dirhash) _at_ /usr/src/sys/ufs/ufs/ufs_dirhash.c:284
KDB: stack
backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame
0xfffffe104b8403b0
kdb_backtrace() at kdb_backtrace+0x39/frame
0xfffffe104b840460
witness_checkorder() at
witness_checkorder+0xd23/frame 0xfffffe104b8404f0
_sx_xlock() at
_sx_xlock+0x75/frame 0xfffffe104b840530
ufsdirhash_add() at
ufsdirhash_add+0x3b/frame 0xfffffe104b840570
ufs_direnter() at
ufs_direnter+0x688/frame 0xfffffe104b840630
ufs_makeinode() at
ufs_makeinode+0x573/frame 0xfffffe104b8407f0
VOP_CREATE_APV() at
VOP_CREATE_APV+0xea/frame 0xfffffe104b840820
vn_open_cred() at
vn_open_cred+0x300/frame 0xfffffe104b840970
kern_openat() at
kern_openat+0x261/frame 0xfffffe104b840ae0
amd64_syscall() at
amd64_syscall+0x265/frame 0xfffffe104b840bf0
Xfast_syscall() at
Xfast_syscall+0xfb/frame 0xfffffe104b840bf0
--- syscall (5, FreeBSD
ELF64, sys_open), rip = 0x801833eca, rsp = 0x7fffffffd768, rbp =
0x7fffffffd7a0 --- 

lock order reversal:
 1st 0xfffff800138ac068 ufs
(ufs) _at_ /usr/src/sys/kern/vfs_subr.c:2101
 2nd 0xfffffe0f9446ff28
bufwait (bufwait) _at_ /usr/src/sys/ufs/ffs/ffs_vnops.c:262
 3rd
0xfffff801c952a068 ufs (ufs) _at_ /usr/src/sys/kern/vfs_subr.c:2101
KDB:
stack backtrace:
db_trace_self_wrapper() at
db_trace_self_wrapper+0x2b/frame 0xfffffe104b83ff20
kdb_backtrace() at
kdb_backtrace+0x39/frame 0xfffffe104b83ffd0
witness_checkorder() at
witness_checkorder+0xd23/frame 0xfffffe104b840060
__lockmgr_args() at
__lockmgr_args+0x878/frame 0xfffffe104b840190
ffs_lock() at
ffs_lock+0x84/frame 0xfffffe104b8401e0
VOP_LOCK1_APV() at
VOP_LOCK1_APV+0xf5/frame 0xfffffe104b840210
_vn_lock() at
_vn_lock+0xab/frame 0xfffffe104b840280
vget() at vget+0x70/frame
0xfffffe104b8402d0
vfs_hash_get() at vfs_hash_get+0xf5/frame
0xfffffe104b840320
ffs_vgetf() at ffs_vgetf+0x41/frame
0xfffffe104b8403b0
softdep_sync_buf() at softdep_sync_buf+0xa8c/frame
0xfffffe104b840490
ffs_syncvnode() at ffs_syncvnode+0x258/frame
0xfffffe104b840510
ffs_fsync() at ffs_fsync+0x20/frame
0xfffffe104b840540
VOP_FSYNC_APV() at VOP_FSYNC_APV+0xf0/frame
0xfffffe104b840570
ufs_direnter() at ufs_direnter+0x5c8/frame
0xfffffe104b840630
ufs_makeinode() at ufs_makeinode+0x573/frame
0xfffffe104b8407f0
VOP_CREATE_APV() at VOP_CREATE_APV+0xea/frame
0xfffffe104b840820
vn_open_cred() at vn_open_cred+0x300/frame
0xfffffe104b840970
kern_openat() at kern_openat+0x261/frame
0xfffffe104b840ae0
amd64_syscall() at amd64_syscall+0x265/frame
0xfffffe104b840bf0
Xfast_syscall() at Xfast_syscall+0xfb/frame
0xfffffe104b840bf0
--- syscall (5, FreeBSD ELF64, sys_open), rip =
0x801833eca, rsp = 0x7fffffffd768, rbp = 0x7fffffffd7a0 ---
lock order
reversal:
 1st 0xfffff80d69ab2418 ufs (ufs) _at_
/usr/src/sys/kern/vfs_mount.c:851
 2nd 0xfffff8033fba5068 devfs (devfs)
_at_ /usr/src/sys/kern/vfs_subr.c:2101
KDB: stack
backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame
0xfffffe104b8723d0
kdb_backtrace() at kdb_backtrace+0x39/frame
0xfffffe104b872480
witness_checkorder() at
witness_checkorder+0xd23/frame 0xfffffe104b872510
__lockmgr_args() at
__lockmgr_args+0x878/frame 0xfffffe104b872640
vop_stdlock() at
vop_stdlock+0x3c/frame 0xfffffe104b872660
VOP_LOCK1_APV() at
VOP_LOCK1_APV+0xf5/frame 0xfffffe104b872690
_vn_lock() at
_vn_lock+0xab/frame 0xfffffe104b872700
vget() at vget+0x70/frame
0xfffffe104b872750
devfs_allocv() at devfs_allocv+0xfd/frame
0xfffffe104b8727a0
devfs_root() at devfs_root+0x43/frame
0xfffffe104b8727d0
vfs_donmount() at vfs_donmount+0x115e/frame
0xfffffe104b872aa0
sys_nmount() at sys_nmount+0x72/frame
0xfffffe104b872ae0
amd64_syscall() at amd64_syscall+0x265/frame
0xfffffe104b872bf0
Xfast_syscall() at Xfast_syscall+0xfb/frame
0xfffffe104b872bf0
--- syscall (378, FreeBSD ELF64, sys_nmount), rip =
0x800a9ecba, rsp = 0x7fffffffcb18, rbp = 0x7fffffffd080 ---


root_at_custom:/usr/local/poudriere/data/logs/bulk/current-default/2014-02-14_11h55m32s
# uname -a
FreeBSD custom 11.0-CURRENT FreeBSD 11.0-CURRENT #0 r261885:
Fri Feb 14 08:51:48 EST 2014 mikej_at_custom:/usr/obj/usr/src/sys/GENERIC
amd64 

Copyright (c) 1992-2014 The FreeBSD Project.
Copyright (c) 1979,
1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
 The Regents of the
University of California. All rights reserved.
FreeBSD is a registered
trademark of The FreeBSD Foundation.
FreeBSD 11.0-CURRENT #0 r261885:
Fri Feb 14 08:51:48 EST 2014
 mikej_at_custom:/usr/obj/usr/src/sys/GENERIC
amd64
FreeBSD clang version 3.3 (tags/RELEASE_33/final 183502)
20130610
WARNING: WITNESS option enabled, expect reduced
performance.
CPU: Intel(R) Xeon(R) CPU X7350 _at_ 2.93GHz (2925.93-MHz
K8-class CPU)
 Origin="GenuineIntel" Id=0x6fb Family=0x6 Model=0xf
Stepping=11
 Features=0xbfebfbff
 Features2=0x4e3bd
 AMD
Features=0x20100800
 AMD Features2=0x1
 TSC: P-state invariant,
performance statistics
real memory = 68719476736 (65536 MB)
avail memory
= 66702168064 (63612 MB)
Event timer "LAPIC" quality 400
ACPI APIC
Table: 
FreeBSD/SMP: Multiprocessor System Detected: 16
CPUs
FreeBSD/SMP: 4 package(s) x 4 core(s)
 cpu0 (BSP): APIC ID: 0
 cpu1
(AP): APIC ID: 1
 cpu2 (AP): APIC ID: 2
 cpu3 (AP): APIC ID: 3
 cpu4
(AP): APIC ID: 8
 cpu5 (AP): APIC ID: 9
 cpu6 (AP): APIC ID: 10
 cpu7
(AP): APIC ID: 11
 cpu8 (AP): APIC ID: 16
 cpu9 (AP): APIC ID: 17
 cpu10
(AP): APIC ID: 18
 cpu11 (AP): APIC ID: 19
 cpu12 (AP): APIC ID: 24

cpu13 (AP): APIC ID: 25
 cpu14 (AP): APIC ID: 26
 cpu15 (AP): APIC ID:
27
ioapic0: Changing APIC ID to 6
ioapic1: Changing APIC ID to 7
ioapic0
irqs 0-23 on motherboard
ioapic1  irqs 32-55 on motherboard
random: 
initialized
kbd1 at kbdmux0
acpi0:  on motherboard
acpi0: Power Button
(fixed)
cpu0:  on acpi0
cpu1:  on acpi0
cpu2:  on acpi0
cpu3:  on
acpi0
cpu4:  on acpi0
cpu5:  on acpi0
cpu6:  on acpi0
cpu7:  on
acpi0
cpu8:  on acpi0 

cpu9:  on acpi0
cpu10:  on acpi0
cpu11:  on
acpi0
cpu12:  on acpi0
cpu13:  on acpi0
cpu14:  on acpi0
cpu15:  on
acpi0
atrtc0:  port 0x70-0x7f irq 8 on acpi0
Event timer "RTC" frequency
32768 Hz quality 0
attimer0:  port 0x40-0x5f irq 0 on acpi0
Timecounter
"i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency
1193182 Hz quality 100
hpet0:  iomem 0xfed00000-0xfed003ff on
acpi0
Timecounter "HPET" frequency 14318180 Hz quality 950
Event timer
"HPET" frequency 14318180 Hz quality 350
Event timer "HPET1" frequency
14318180 Hz quality 340
Event timer "HPET2" frequency 14318180 Hz
quality 340
Timecounter "ACPI-fast" frequency 3579545 Hz quality
900
acpi_timer0:  port 0x808-0x80b on acpi0
pcib0:  port 0xcf8-0xcff on
acpi0
pci0:  on pcib0
pcib1:  irq 36 at device 1.0 on pci0
pci20:  on
pcib1
pcib2:  irq 17 at device 2.0 on pci0
pci22:  on pcib2
pcib3:  irq
17 at device 0.0 on pci22
pci23:  on pcib3
pcib4:  irq 17 at device 0.0
on pci23
pci25:  on pcib4
mfi0:  port 0xec00-0xecff mem
0xde380000-0xde3bffff,0xde3c0000-0xde3fffff irq 17 at device 0.0 on
pci25
mfi0: Using MSI
mfi0: Megaraid SAS driver Ver 4.23 
mfi0: FW
MaxCmds = 1008, limiting to 128
pcib5:  irq 18 at device 1.0 on
pci23
pci24:  on pcib5
pcib6:  at device 0.3 on pci22
pci26:  on
pcib6
pcib7:  irq 37 at device 3.0 on pci0
pci1:  on pcib7
pcib8:  at
device 0.0 on pci1
pci2:  on pcib8
pcib9:  at device 1.0 on pci2
pci3: 
on pcib9
pcib10:  at device 2.0 on pci2
pci10:  on pcib10
pcib11:  at
device 0.0 on pci10
pci11:  on pcib11
bce0:  mem 0xd6000000-0xd7ffffff
irq 18 at device 0.0 on pci11
miibus0:  on bce0
brgphy0:  PHY 1 on
miibus0
brgphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX,
1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto,
auto-flow
bce0: Ethernet address: a4:ba:db:34:77:df
bce0: ASIC
(0x57081020); Rev (B2); Bus (PCI-X, 64-bit, 133MHz); B/C (7.4.0); Bufs
(RX:2;TX:2;PG:8); Flags (SPLT|MSI|MFW); MFW (ipms 1.6.0) 

Coal
(RX:6,6,18,18; TX:20,20,80,80)
pcib12:  at device 3.0 on pci2
pci8:  on
pcib12
pcib13:  at device 0.0 on pci8
pci9:  on pcib13
bce1:  mem
0xd8000000-0xd9ffffff irq 19 at device 0.0 on pci9
miibus1:  on
bce1
brgphy1:  PHY 1 on miibus1
brgphy1: 10baseT, 10baseT-FDX,
100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX,
1000baseT-FDX-master, auto, auto-flow
bce1: Ethernet address:
a4:ba:db:34:77:dd
bce1: ASIC (0x57081020); Rev (B2); Bus (PCI-X, 64-bit,
133MHz); B/C (7.4.0); Bufs (RX:2;TX:2;PG:8); Flags (SPLT|MSI|MFW); MFW
(ipms 1.6.0)
Coal (RX:6,6,18,18; TX:20,20,80,80)
pcib14:  at device 4.0
on pci2
pci6:  on pcib14
pcib15:  at device 0.0 on pci6
pci7:  on
pcib15
bce2:  mem 0xda000000-0xdbffffff irq 37 at device 0.0 on
pci7
miibus2:  on bce2
brgphy2:  PHY 1 on miibus2
brgphy2: 10baseT,
10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master,
1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow
bce2: Ethernet
address: a4:ba:db:34:77:db
bce2: ASIC (0x57081020); Rev (B2); Bus
(PCI-X, 64-bit, 133MHz); B/C (7.4.0); Bufs (RX:2;TX:2;PG:8); Flags
(SPLT|MSI|MFW); MFW (UMP 1.1.9)
Coal (RX:6,6,18,18;
TX:20,20,80,80)
pcib16:  at device 5.0 on pci2
pci4:  on pcib16
pcib17: 
at device 0.0 on pci4
pci5:  on pcib17
bce3:  mem 0xdc000000-0xddffffff
irq 38 at device 0.0 on pci5
miibus3:  on bce3
brgphy3:  PHY 1 on
miibus3
brgphy3: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX,
1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto,
auto-flow
bce3: Ethernet address: a4:ba:db:34:77:d9
bce3: ASIC
(0x57081020); Rev (B2); Bus (PCI-X, 64-bit, 133MHz); B/C (7.4.0); Bufs
(RX:2;TX:2;PG:8); Flags (SPLT|MSI|MFW); MFW (UMP 1.1.9)
Coal
(RX:6,6,18,18; TX:20,20,80,80)
pcib18:  irq 32 at device 4.0 on
pci0
pci12:  on pcib18
pcib19:  at device 0.0 on pci12
pci13:  on
pcib19
pcib20:  at device 2.0 on pci13
pci15:  on pcib20
pcib21:  at
device 4.0 on pci13
pci14:  on pcib21
pcib22:  irq 33 at device 6.0 on
pci0
pci16:  on pcib22
pcib23:  at device 0.0 on pci16
pci17:  on
pcib23
pcib24:  at device 2.0 on pci17
pci19:  on pcib24
pcib25:  at
device 4.0 on pci17
pci18:  on pcib25
pci0:  at device 8.0 (no driver
attached)
pcib26:  at device 28.0 on pci0
Coal (RX:6,6,18,18;
TX:20,20,80,80)
pcib12:  at device 3.0 on pci2
pci8:  on pcib12
pcib13: 
at device 0.0 on pci8
pci9:  on pcib13
bce1:  mem 0xd8000000-0xd9ffffff
irq 19 at device 0.0 on pci9
miibus1:  on bce1
brgphy1:  PHY 1 on
miibus1
brgphy1: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX,
1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto,
auto-flow
bce1: Ethernet address: a4:ba:db:34:77:dd
bce1: ASIC
(0x57081020); Rev (B2); Bus (PCI-X, 64-bit, 133MHz); B/C (7.4.0); Bufs
(RX:2;TX:2;PG:8); Flags (SPLT|MSI|MFW); MFW (ipms 1.6.0)
Coal
(RX:6,6,18,18; TX:20,20,80,80)
pcib14:  at device 4.0 on pci2
pci6:  on
pcib14
pcib15:  at device 0.0 on pci6
pci7:  on pcib15
bce2:  mem
0xda000000-0xdbffffff irq 37 at device 0.0 on pci7
miibus2:  on
bce2
brgphy2:  PHY 1 on miibus2
brgphy2: 10baseT, 10baseT-FDX,
100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX,
1000baseT-FDX-master, auto, auto-flow
bce2: Ethernet address:
a4:ba:db:34:77:db
bce2: ASIC (0x57081020); Rev (B2); Bus (PCI-X, 64-bit,
133MHz); B/C (7.4.0); Bufs (RX:2;TX:2;PG:8); Flags (SPLT|MSI|MFW); MFW
(UMP 1.1.9)
Coal (RX:6,6,18,18; TX:20,20,80,80)
pcib16:  at device 5.0
on pci2
pci4:  on pcib16
pcib17:  at device 0.0 on pci4
pci5:  on
pcib17
bce3:  mem 0xdc000000-0xddffffff irq 38 at device 0.0 on
pci5
miibus3:  on bce3
brgphy3:  PHY 1 on miibus3
brgphy3: 10baseT,
10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master,
1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow
bce3: Ethernet
address: a4:ba:db:34:77:d9
bce3: ASIC (0x57081020); Rev (B2); Bus
(PCI-X, 64-bit, 133MHz); B/C (7.4.0); Bufs (RX:2;TX:2;PG:8); Flags
(SPLT|MSI|MFW); MFW (UMP 1.1.9)
Coal (RX:6,6,18,18;
TX:20,20,80,80)
pcib18:  irq 32 at device 4.0 on pci0
pci12:  on
pcib18
pcib19:  at device 0.0 on pci12
pci13:  on pcib19
pcib20:  at
device 2.0 on pci13
pci15:  on pcib20
pcib21:  at device 4.0 on
pci13
pci14:  on pcib21
pcib22:  irq 33 at device 6.0 on pci0
pci16:  on
pcib22
pcib23:  at device 0.0 on pci16
pci17:  on pcib23
pcib24:  at
device 2.0 on pci17
pci19:  on pcib24
pcib25:  at device 4.0 on
pci17
pci18:  on pcib25
pci0:  at device 8.0 (no driver
attached)
pcib26:  at device 28.0 on pci0 

Coal (RX:6,6,18,18;
TX:20,20,80,80)
pcib12:  at device 3.0 on pci2
pci8:  on pcib12
pcib13: 
at device 0.0 on pci8
pci9:  on pcib13
bce1:  mem 0xd8000000-0xd9ffffff
irq 19 at device 0.0 on pci9
miibus1:  on bce1
brgphy1:  PHY 1 on
miibus1
brgphy1: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX,
1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto,
auto-flow
bce1: Ethernet address: a4:ba:db:34:77:dd
bce1: ASIC
(0x57081020); Rev (B2); Bus (PCI-X, 64-bit, 133MHz); B/C (7.4.0); Bufs
(RX:2;TX:2;PG:8); Flags (SPLT|MSI|MFW); MFW (ipms 1.6.0)
Coal
(RX:6,6,18,18; TX:20,20,80,80)
pcib14:  at device 4.0 on pci2
pci6:  on
pcib14
pcib15:  at device 0.0 on pci6
pci7:  on pcib15
bce2:  mem
0xda000000-0xdbffffff irq 37 at device 0.0 on pci7
miibus2:  on
bce2
brgphy2:  PHY 1 on miibus2
brgphy2: 10baseT, 10baseT-FDX,
100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX,
1000baseT-FDX-master, auto, auto-flow
bce2: Ethernet address:
a4:ba:db:34:77:db
bce2: ASIC (0x57081020); Rev (B2); Bus (PCI-X, 64-bit,
133MHz); B/C (7.4.0); Bufs (RX:2;TX:2;PG:8); Flags (SPLT|MSI|MFW); MFW
(UMP 1.1.9)
Coal (RX:6,6,18,18; TX:20,20,80,80)
pcib16:  at device 5.0
on pci2
pci4:  on pcib16
pcib17:  at device 0.0 on pci4
pci5:  on
pcib17
bce3:  mem 0xdc000000-0xddffffff irq 38 at device 0.0 on
pci5
miibus3:  on bce3
brgphy3:  PHY 1 on miibus3
brgphy3: 10baseT,
10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master,
1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow
bce3: Ethernet
address: a4:ba:db:34:77:d9
bce3: ASIC (0x57081020); Rev (B2); Bus
(PCI-X, 64-bit, 133MHz); B/C (7.4.0); Bufs (RX:2;TX:2;PG:8); Flags
(SPLT|MSI|MFW); MFW (UMP 1.1.9)
Coal (RX:6,6,18,18;
TX:20,20,80,80)
pcib18:  irq 32 at device 4.0 on pci0
pci12:  on
pcib18
pcib19:  at device 0.0 on pci12
pci13:  on pcib19
pcib20:  at
device 2.0 on pci13
pci15:  on pcib20
pcib21:  at device 4.0 on
pci13
pci14:  on pcib21
pcib22:  irq 33 at device 6.0 on pci0
pci16:  on
pcib22
pcib23:  at device 0.0 on pci16
pci17:  on pcib23
pcib24:  at
device 2.0 on pci17
pci19:  on pcib24
pcib25:  at device 4.0 on
pci17
pci18:  on pcib25
pci0:  at device 8.0 (no driver
attached)
pcib26:  at device 28.0 on pci0 

Coal (RX:6,6,18,18;
TX:20,20,80,80)
pcib12:  at device 3.0 on pci2
pci8:  on pcib12
pcib13: 
at device 0.0 on pci8
pci9:  on pcib13
bce1:  mem 0xd8000000-0xd9ffffff
irq 19 at device 0.0 on pci9
miibus1:  on bce1
brgphy1:  PHY 1 on
miibus1
brgphy1: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX,
1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto,
auto-flow
bce1: Ethernet address: a4:ba:db:34:77:dd
bce1: ASIC
(0x57081020); Rev (B2); Bus (PCI-X, 64-bit, 133MHz); B/C (7.4.0); Bufs
(RX:2;TX:2;PG:8); Flags (SPLT|MSI|MFW); MFW (ipms 1.6.0)
Coal
(RX:6,6,18,18; TX:20,20,80,80)
pcib14:  at device 4.0 on pci2
pci6:  on
pcib14
pcib15:  at device 0.0 on pci6
pci7:  on pcib15
bce2:  mem
0xda000000-0xdbffffff irq 37 at device 0.0 on pci7
miibus2:  on
bce2
brgphy2:  PHY 1 on miibus2
brgphy2: 10baseT, 10baseT-FDX,
100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX,
1000baseT-FDX-master, auto, auto-flow
bce2: Ethernet address:
a4:ba:db:34:77:db
bce2: ASIC (0x57081020); Rev (B2); Bus (PCI-X, 64-bit,
133MHz); B/C (7.4.0); Bufs (RX:2;TX:2;PG:8); Flags (SPLT|MSI|MFW); MFW
(UMP 1.1.9)
Coal (RX:6,6,18,18; TX:20,20,80,80)
pcib16:  at device 5.0
on pci2
pci4:  on pcib16
pcib17:  at device 0.0 on pci4
pci5:  on
pcib17
bce3:  mem 0xdc000000-0xddffffff irq 38 at device 0.0 on
pci5
miibus3:  on bce3
brgphy3:  PHY 1 on miibus3
brgphy3: 10baseT,
10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master,
1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow
bce3: Ethernet
address: a4:ba:db:34:77:d9
bce3: ASIC (0x57081020); Rev (B2); Bus
(PCI-X, 64-bit, 133MHz); B/C (7.4.0); Bufs (RX:2;TX:2;PG:8); Flags
(SPLT|MSI|MFW); MFW (UMP 1.1.9)
Coal (RX:6,6,18,18;
TX:20,20,80,80)
pcib18:  irq 32 at device 4.0 on pci0
pci12:  on
pcib18
pcib19:  at device 0.0 on pci12
pci13:  on pcib19
pcib20:  at
device 2.0 on pci13
pci15:  on pcib20
pcib21:  at device 4.0 on
pci13
pci14:  on pcib21
pcib22:  irq 33 at device 6.0 on pci0
pci16:  on
pcib22
pcib23:  at device 0.0 on pci16
pci17:  on pcib23
pcib24:  at
device 2.0 on pci17
pci19:  on pcib24
pcib25:  at device 4.0 on
pci17
pci18:  on pcib25
pci0:  at device 8.0 (no driver
attached)
pcib26:  at device 28.0 on pci0 

Coal (RX:6,6,18,18;
TX:20,20,80,80)
pcib12:  at device 3.0 on pci2
pci8:  on pcib12
pcib13: 
at device 0.0 on pci8
pci9:  on pcib13
bce1:  mem 0xd8000000-0xd9ffffff
irq 19 at device 0.0 on pci9
miibus1:  on bce1
brgphy1:  PHY 1 on
miibus1
brgphy1: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX,
1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto,
auto-flow
bce1: Ethernet address: a4:ba:db:34:77:dd
bce1: ASIC
(0x57081020); Rev (B2); Bus (PCI-X, 64-bit, 133MHz); B/C (7.4.0); Bufs
(RX:2;TX:2;PG:8); Flags (SPLT|MSI|MFW); MFW (ipms 1.6.0)
Coal
(RX:6,6,18,18; TX:20,20,80,80)
pcib14:  at device 4.0 on pci2
pci6:  on
pcib14
pcib15:  at device 0.0 on pci6
pci7:  on pcib15
bce2:  mem
0xda000000-0xdbffffff irq 37 at device 0.0 on pci7
miibus2:  on
bce2
brgphy2:  PHY 1 on miibus2
brgphy2: 10baseT, 10baseT-FDX,
100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX,
1000baseT-FDX-master, auto, auto-flow
bce2: Ethernet address:
a4:ba:db:34:77:db
bce2: ASIC (0x57081020); Rev (B2); Bus (PCI-X, 64-bit,
133MHz); B/C (7.4.0); Bufs (RX:2;TX:2;PG:8); Flags (SPLT|MSI|MFW); MFW
(UMP 1.1.9)
Coal (RX:6,6,18,18; TX:20,20,80,80)
pcib16:  at device 5.0
on pci2
pci4:  on pcib16
pcib17:  at device 0.0 on pci4
pci5:  on
pcib17
bce3:  mem 0xdc000000-0xddffffff irq 38 at device 0.0 on
pci5
miibus3:  on bce3
brgphy3:  PHY 1 on miibus3
brgphy3: 10baseT,
10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master,
1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow
bce3: Ethernet
address: a4:ba:db:34:77:d9
bce3: ASIC (0x57081020); Rev (B2); Bus
(PCI-X, 64-bit, 133MHz); B/C (7.4.0); Bufs (RX:2;TX:2;PG:8); Flags
(SPLT|MSI|MFW); MFW (UMP 1.1.9)
Coal (RX:6,6,18,18;
TX:20,20,80,80)
pcib18:  irq 32 at device 4.0 on pci0
pci12:  on
pcib18
pcib19:  at device 0.0 on pci12
pci13:  on pcib19
pcib20:  at
device 2.0 on pci13
pci15:  on pcib20
pcib21:  at device 4.0 on
pci13
pci14:  on pcib21
pcib22:  irq 33 at device 6.0 on pci0
pci16:  on
pcib22
pcib23:  at device 0.0 on pci16
pci17:  on pcib23
pcib24:  at
device 2.0 on pci17
pci19:  on pcib24
pcib25:  at device 4.0 on
pci17
pci18:  on pcib25
pci0:  at device 8.0 (no driver
attached)
pcib26:  at device 28.0 on pci0 

Coal (RX:6,6,18,18;
TX:20,20,80,80)
pcib12:  at device 3.0 on pci2
pci8:  on pcib12
pcib13: 
at device 0.0 on pci8
pci9:  on pcib13
bce1:  mem 0xd8000000-0xd9ffffff
irq 19 at device 0.0 on pci9
miibus1:  on bce1
brgphy1:  PHY 1 on
miibus1
brgphy1: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX,
1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto,
auto-flow
bce1: Ethernet address: a4:ba:db:34:77:dd
bce1: ASIC
(0x57081020); Rev (B2); Bus (PCI-X, 64-bit, 133MHz); B/C (7.4.0); Bufs
(RX:2;TX:2;PG:8); Flags (SPLT|MSI|MFW); MFW (ipms 1.6.0)
Coal
(RX:6,6,18,18; TX:20,20,80,80)
pcib14:  at device 4.0 on pci2
pci6:  on
pcib14
pcib15:  at device 0.0 on pci6
pci7:  on pcib15
bce2:  mem
0xda000000-0xdbffffff irq 37 at device 0.0 on pci7
miibus2:  on
bce2
brgphy2:  PHY 1 on miibus2
brgphy2: 10baseT, 10baseT-FDX,
100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX,
1000baseT-FDX-master, auto, auto-flow
bce2: Ethernet address:
a4:ba:db:34:77:db
bce2: ASIC (0x57081020); Rev (B2); Bus (PCI-X, 64-bit,
133MHz); B/C (7.4.0); Bufs (RX:2;TX:2;PG:8); Flags (SPLT|MSI|MFW); MFW
(UMP 1.1.9)
Coal (RX:6,6,18,18; TX:20,20,80,80)
pcib16:  at device 5.0
on pci2
pci4:  on pcib16
pcib17:  at device 0.0 on pci4
pci5:  on
pcib17
bce3:  mem 0xdc000000-0xddffffff irq 38 at device 0.0 on
pci5
miibus3:  on bce3
brgphy3:  PHY 1 on miibus3
brgphy3: 10baseT,
10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master,
1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow
bce3: Ethernet
address: a4:ba:db:34:77:d9
bce3: ASIC (0x57081020); Rev (B2); Bus
(PCI-X, 64-bit, 133MHz); B/C (7.4.0); Bufs (RX:2;TX:2;PG:8); Flags
(SPLT|MSI|MFW); MFW (UMP 1.1.9)
Coal (RX:6,6,18,18;
TX:20,20,80,80)
pcib18:  irq 32 at device 4.0 on pci0
pci12:  on
pcib18
pcib19:  at device 0.0 on pci12
pci13:  on pcib19
pcib20:  at
device 2.0 on pci13
pci15:  on pcib20
pcib21:  at device 4.0 on
pci13
pci14:  on pcib21
pcib22:  irq 33 at device 6.0 on pci0
pci16:  on
pcib22
pcib23:  at device 0.0 on pci16
pci17:  on pcib23
pcib24:  at
device 2.0 on pci17
pci19:  on pcib24
pcib25:  at device 4.0 on
pci17
pci18:  on pcib25
pci0:  at device 8.0 (no driver
attached)
pcib26:  at device 28.0 on pci0