FreeBSD 7.0Beta4 (amd64) freezes with dummynet enabled

From: Thomas Vogt <freebsdlists_at_bsdunix.ch>
Date: Mon, 10 Dec 2007 16:36:23 +0100
Hi,

I kindly ask for advice for debugging a problem with 7.0-Beta 4 (amd64).

The machine freezes if dummynet rules are enabled on a gigabit port. The
system runs fine on fast ethernet or with dummynet rules disabled. I
can't trigger the bug if my upstream port is limited to 100mbit. The
machine suddenly freezes after 3-4 minutes if dummynet rules are loaded.

My Kernel already includes some debug options but I'm not able to
produce a panic.

Kernel:
options         IPFIREWALL
options         IPFIREWALL_VERBOSE
options         IPFIREWALL_VERBOSE_LIMIT=100
options         IPFIREWALL_DEFAULT_TO_ACCEPT
options         DUMMYNET
options         ZERO_COPY_SOCKETS
options         DEVICE_POLLING
options         HZ=1000

options     INVARIANTS
options     INVARIANT_SUPPORT
options     WITNESS
options     DEBUG_VFS_LOCKS
options     DIAGNOSTIC
options KDB
options KDB_UNATTENDED
options DDB
options GDB
options KDB_TRACE
options SOCKBUF_DEBUG
options BREAK_TO_DEBUGGER

dmesg:
Copyright (c) 1992-2007 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 7.0-BETA4 #4: Fri Dec  7 12:45:37 UTC 2007
    root_at_lisa.mlan.solnet.ch:/usr/obj/usr/src/sys/SOLNETDEBUG
WARNING: WITNESS option enabled, expect reduced performance.
WARNING: DIAGNOSTIC option enabled, expect reduced performance.
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel(R) Xeon(R) CPU            5110  _at_ 1.60GHz (1739.55-MHz
K8-class CPU)
  Origin = "GenuineIntel"  Id = 0x6f6  Stepping = 6

Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x4e33d<SSE3,RSVD2,MON,DS_CPL,VMX,TM2,SSSE3,CX16,xTPR,PDCM,DCA>
  AMD Features=0x20100800<SYSCALL,NX,LM>
  AMD Features2=0x1<LAHF>
  Cores per package: 2
usable memory = 4258033664 (4060 MB)
avail memory  = 4071751680 (3883 MB)
ACPI APIC Table: <INTEL  S5000XVN>
FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
 cpu0 (BSP): APIC ID:  0
 cpu1 (AP): APIC ID:  1
WITNESS: spin lock intrcnt not in order list
ioapic0 <Version 2.0> irqs 0-23 on motherboard
ioapic1 <Version 2.0> irqs 24-47 on motherboard
lapic0: Forcing LINT1 to edge trigger
kbd1 at kbdmux0
acpi0: <INTEL S5000XVN> on motherboard
acpi0: [ITHREAD]
acpi0: Power Button (fixed)
acpi0: reservation of 0, a0000 (3) failed
Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
acpi_hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on
acpi0
Timecounter "HPET" frequency 14318180 Hz quality 900
cpu0: <ACPI CPU> on acpi0
device_attach: acpi_perf0 attach returned 6
device_attach: acpi_perf0 attach returned 6
p4tcc0: <CPU Frequency Thermal Control> on cpu0
cpu1: <ACPI CPU> on acpi0
device_attach: acpi_perf1 attach returned 6
device_attach: acpi_perf1 attach returned 6
p4tcc1: <CPU Frequency Thermal Control> on cpu1
acpi_button0: <Sleep Button> on acpi0
acpi_button1: <Power Button> on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xca2,0xca3,0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> at device 2.0 on pci0
pci1: <ACPI PCI bus> on pcib1
pcib2: <ACPI PCI-PCI bridge> irq 16 at device 0.0 on pci1
pci2: <ACPI PCI bus> on pcib2
pcib3: <ACPI PCI-PCI bridge> irq 16 at device 0.0 on pci2
pci3: <ACPI PCI bus> on pcib3
pcib4: <PCI-PCI bridge> at device 0.0 on pci3
pci4: <PCI bus> on pcib4
arcmsr0: <Areca SATA Host Adapter RAID Controller (RAID6 capable)
> mem 0xc3d00000-0xc3d00fff,0xc3000000-0xc33fffff irq 18 at device 14.0
on pci4
ARECA RAID ADAPTER0: Driver Version 1.20.00.14 2007-2-05
ARECA RAID ADAPTER0: FIRMWARE VERSION V1.43 2007-4-17
arcmsr0: [ITHREAD]
pcib5: <PCI-PCI bridge> at device 0.2 on pci3
pci5: <PCI bus> on pcib5
pcib6: <ACPI PCI-PCI bridge> irq 18 at device 2.0 on pci2
pci6: <ACPI PCI bus> on pcib6
em0: <Intel(R) PRO/1000 Network Connection Version - 6.7.3> port
0x2020-0x203f mem 0xc3c20000-0xc3c3ffff,0xc3800000-0xc3bfffff irq 18 at
device 0.0 on pci6
em0: Using MSI interrupt
em0: Ethernet address: 00:15:17:30:94:30
em0: [FILTER]
em1: <Intel(R) PRO/1000 Network Connection Version - 6.7.3> port
0x2000-0x201f mem 0xc3c00000-0xc3c1ffff,0xc3400000-0xc37fffff irq 19 at
device 0.1 on pci6
em1: Using MSI interrupt
em1: Ethernet address: 00:15:17:30:94:31
em1: [FILTER]
pcib7: <ACPI PCI-PCI bridge> at device 0.3 on pci1
pci7: <ACPI PCI bus> on pcib7
pcib8: <PCI-PCI bridge> at device 3.0 on pci0
pci8: <PCI bus> on pcib8
pcib9: <ACPI PCI-PCI bridge> at device 4.0 on pci0
pci9: <ACPI PCI bus> on pcib9
vgapci0: <VGA-compatible display> port 0x1000-0x107f mem
0xc2000000-0xc2ffffff,0xb0000000-0xbfffffff,0xc0000000-0xc1ffffff irq 16
at device 0.0 on pci9
pcib10: <ACPI PCI-PCI bridge> at device 5.0 on pci0
pci10: <ACPI PCI bus> on pcib10
pcib11: <ACPI PCI-PCI bridge> at device 6.0 on pci0
pci11: <ACPI PCI bus> on pcib11
pcib12: <PCI-PCI bridge> at device 7.0 on pci0
pci12: <PCI bus> on pcib12
pci0: <base peripheral> at device 8.0 (no driver attached)
pci0: <multimedia> at device 27.0 (no driver attached)
pcib13: <ACPI PCI-PCI bridge> irq 16 at device 28.0 on pci0
pci13: <ACPI PCI bus> on pcib13
uhci0: <Intel 631XESB/632XESB/3100 USB controller USB-1> port
0x3080-0x309f irq 23 at device 29.0 on pci0
uhci0: [GIANT-LOCKED]
uhci0: [ITHREAD]
usb0: <Intel 631XESB/632XESB/3100 USB controller USB-1> on uhci0
usb0: USB revision 1.0
uhub0: <Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb0
uhub0: 2 ports with 2 removable, self powered
uhci1: <Intel 631XESB/632XESB/3100 USB controller USB-2> port
0x3060-0x307f irq 22 at device 29.1 on pci0
uhci1: [GIANT-LOCKED]
uhci1: [ITHREAD]
usb1: <Intel 631XESB/632XESB/3100 USB controller USB-2> on uhci1
usb1: USB revision 1.0
uhub1: <Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb1
uhub1: 2 ports with 2 removable, self powered
uhci2: <Intel 631XESB/632XESB/3100 USB controller USB-3> port
0x3040-0x305f irq 23 at device 29.2 on pci0
uhci2: [GIANT-LOCKED]
uhci2: [ITHREAD]
usb2: <Intel 631XESB/632XESB/3100 USB controller USB-3> on uhci2
usb2: USB revision 1.0
uhub2: <Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb2
uhub2: 2 ports with 2 removable, self powered
uhci3: <Intel 631XESB/632XESB/3100 USB controller USB-4> port
0x3020-0x303f irq 22 at device 29.3 on pci0
uhci3: [GIANT-LOCKED]
uhci3: [ITHREAD]
usb3: <Intel 631XESB/632XESB/3100 USB controller USB-4> on uhci3
usb3: USB revision 1.0
uhub3: <Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb3
uhub3: 2 ports with 2 removable, self powered
ehci0: <Intel 63XXESB USB 2.0 controller> mem 0xc3f04400-0xc3f047ff irq
23 at device 29.7 on pci0
ehci0: [GIANT-LOCKED]
ehci0: [ITHREAD]
usb4: EHCI version 1.0
usb4: companion controllers, 2 ports each: usb0 usb1 usb2 usb3
usb4: <Intel 63XXESB USB 2.0 controller> on ehci0
usb4: USB revision 2.0
uhub4: <Intel EHCI root hub, class 9/0, rev 2.00/1.00, addr 1> on usb4
uhub4: 8 ports with 8 removable, self powered
pcib14: <ACPI PCI-PCI bridge> at device 30.0 on pci0
pci14: <ACPI PCI bus> on pcib14
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel 63XXESB2 UDMA100 controller> port
0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x30b0-0x30bf irq 20 at device 31.1
on pci0
ata0: <ATA channel 0> on atapci0
ata0: [ITHREAD]
ata1: <ATA channel 1> on atapci0
ata1: [ITHREAD]
atapci1: <Intel 63XXESB2 SATA300 controller> port
0x30c8-0x30cf,0x30e4-0x30e7,0x30c0-0x30c7,0x30e0-0x30e3,0x30a0-0x30af
mem 0xc3f04000-0xc3f043ff irq 20 at device 31.2 on pci0
atapci1: [ITHREAD]
ata2: <ATA channel 0> on atapci1
ata2: [ITHREAD]
ata3: <ATA channel 1> on atapci1
ata3: [ITHREAD]
pci0: <serial bus, SMBus> at device 31.3 (no driver attached)
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
atkbd0: [ITHREAD]
sio0: configured irq 4 not in bitmap of probed irqs 0
sio0: port may not be enabled
sio0: configured irq 4 not in bitmap of probed irqs 0
sio0: port may not be enabled
sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 flags 0x10 on
acpi0
sio0: type 16550A, console
sio0: [FILTER]
sio1: configured irq 3 not in bitmap of probed irqs 0
sio1: port may not be enabled
sio1: configured irq 3 not in bitmap of probed irqs 0
sio1: port may not be enabled
sio1: <16550A-compatible COM port> port 0x2f8-0x2ff irq 3 on acpi0
sio1: type 16550A
sio1: [FILTER]
orm0: <ISA Option ROMs> at iomem
0xcd000-0xcdfff,0xce000-0xcefff,0xcf000-0xcffff on isa0
ppc0: cannot reserve I/O port range
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
Timecounters tick every 1.000 msec
ipfw2 (+ipv6) initialized, divert loadable, rule-based forwarding
disabled, default to accept, logging limited to 100 packets/entry by default
Waiting 5 seconds for SCSI devices to settle
acd0: DVDROM <HL-DT-STDVD-ROM GDR8161B/0100> at ata0-master UDMA33
da0 at arcmsr0 bus 0 target 0 lun 0
da0: <Areca ARC-1230-VOL#00 R001> Fixed Direct Access SCSI-5 device
da0: 166.666MB/s transfers (83.333MHz DT, offset 32, 16bit)
da0: 238418MB (488281088 512 byte sectors: 255H 63S/T 30394C)
da1 at arcmsr0 bus 0 target 0 lun 1
da1: <Seagate ST3750640NS R001> Fixed Direct Access SCSI-5 device
da1: 166.666MB/s transfers (83.333MHz DT, offset 32, 16bit)
da1: 715404MB (1465149168 512 byte sectors: 255H 63S/T 91201C)
da2 at arcmsr0 bus 0 target 0 lun 2
da2: <Seagate ST3750640NS R001> Fixed Direct Access SCSI-5 device
da2: 166.666MB/s transfers (83.333MHz DT, offset 32, 16bit)
da2: 715404MB (1465149168 512 byte sectors: 255H 63S/T 91201C)
da3 at arcmsr0 bus 0 target 0 lun 3
da3: <Seagate ST3750640NS R001> Fixed Direct Access SCSI-5 device
da3: 166.666MB/s transfers (83.333MHz DT, offset 32, 16bit)
da3: 715404MB (1465149168 512 byte sectors: 255H 63S/T 91201C)
da4 at arcmsr0 bus 0 target 0 lun 4
da4: <Seagate ST3750640NS R001> Fixed Direct Access SCSI-5 device
da4: 166.666MB/s transfers (83.333MHz DT, offset 32, 16bit)
da4: 715404MB (1465149168 512 byte sectors: 255H 63S/T 91201C)
da5 at arcmsr0 bus 0 target 0 lun 5
da5: <Seagate ST3750640NS R001> Fixed Direct Access SCSI-5 device
da5: 166.666MB/s transfers (83.333MHz DT, offset 32, 16bit)
da5: 715404MB (1465149168 512 byte sectors: 255H 63S/T 91201C)
da6 at arcmsr0 bus 0 target 0 lun 6
da6: <Seagate ST3750640NS R001> Fixed Direct Access SCSI-5 device
da6: 166.666MB/s transfers (83.333MHz DT, offset 32, 16bit)
da6: 715404MB (1465149168 512 byte sectors: 255H 63S/T 91201C)
da7 at arcmsr0 bus 0 target 0 lun 7
da7: <Seagate ST3750640NS R001> Fixed Direct Access SCSI-5 device
da7: 166.666MB/s transfers (83.333MHz DT, offset 32, 16bit)
da7: 715404MB (1465149168 512 byte sectors: 255H 63S/T 91201C)
lapic1: Forcing LINT1 to edge trigger
SMP: AP CPU #1 Launched!
WARNING: WITNESS option enabled, expect reduced performance.
WARNING: DIAGNOSTIC option enabled, expect reduced performance.
Trying to mount root from ufs:/dev/da0s1a
WARNING: ZFS is considered to be an experimental feature in FreeBSD.
ZFS filesystem version 6
ZFS storage pool version 6
em1: Hardware Initialization Failed
em1: Unable to initialize the hardware
Expensive timeout(9) function: 0xffffffff80258fe0(0xffffff00012fc000)
0.008566172 s
em1: link state changed to UP
em1: link state changed to DOWN
em1: link state changed to UP
Expensive timeout(9) function: 0xffffffff80507ae0(0) 0.105003251 s
Expensive timeout(9) function: 0xffffffff80397260(0) 0.372032111 s
Expensive timeout(9) function: 0xffffffff80469350(0xffffff0069b5f888)
0.403952336 s


my ipfw rules:
pipe 2 config bw 600Mbit/s queue 50KBytes
queue 201 config pipe 2 weight 1 mask dst-ip 0x0000000f queue 50
queue 210 config pipe 2 weight 50 mask dst-ip 0x0000000f queue 50


add 100 permit ip  from any to any via lo0
add 200 deny ip from any to 127.0.0.0/8

# Speedup Established
add 5001 skipto 8000 ip from 82.220.2.0/24 to 212.101.0.0/19 via em1
add 6000 queue 210 ip from any to 212.101.0.0/19 out xmit em1
add 6001 skipto 8000 ip from any to 212.101.0.0/19 out xmit em1
add 6002 queue 210 ip from any to 212.41.65.0/18 out xmit em1
add 6003 skipto 8000 ip from any to 212.41.65.0/18 out xmit em1
add 6004 queue 210 ip from any to 82.220.0.0/18 out xmit em1
add 6005 skipto 8000 ip from any to 82.220.0.0/18 out xmit em1
add 6100 queue 201 ip from any to any out xmit em1

add 8000 permit tcp from any to any via em1 established

# ICMP
add 11700 permit icmp from any to any via em1 icmptype 0,3,4,8,11,12

# Traceroute
add 1800 allow udp from any 30000-60000 to me 30000-60000 via em0


# SSH (external connections are allowed for portsnap maintainer)
add 11900 allow ip from me to any ssh setup
add 11901 allow ip from any to me ssh setup

# HTTP OUT
add 11910 permit tcp from me to any 80 setup via em1

# SMTP
add 12100 permit tcp from me to 212.101.0.0/19 25 setup via em1
add 12110 allow tcp from me to any dst-port smtp via em1 setup

# DNS UDP SolNet
add 21100 permit udp from 212.101.0.10 domain to me recv em1
add 21101 permit udp from me to 212.101.0.10 domain out xmit em1
add 21110 permit udp from 212.101.4.253 domain to me recv em1
add 21111 permit udp from me to 212.101.4.253 domain out xmit em1
add 21120 permit udp from 212.101.4.251 domain to me recv em1
add 21121 permit udp from me to 212.101.4.251 domain out xmit em1

# NTP
add 21440 permit udp from me ntp to 212.101.0.0/19 ntp out xmit em1
add 21445 permit udp from 224.0.1.1 ntp to 212.101.0.0/19 ntp out xmit em1
add 21450 permit udp from 212.101.0.0/19 ntp to me ntp recv em1
add 21460 permit udp from 212.101.0.0/19 ntp to 224.0.1.1 ntp recv em1
add 21470 permit udp from me ntp to 224.0.1.1 ntp out xmit em1


# WEB
add 22250 permit tcp from any to 212.101.4.241 80 setup via em1 limit
src-addr 4
add 22251 permit tcp from any to 212.101.4.241 443 setup via em1 limit
src-addr 4
add 22253 permit tcp from 212.101.4.241 to any 1024-65535 setup via em1


# FTP
add 22300 permit tcp from any to 212.101.4.244 21 setup via em1 limit
src-addr 4
add 22301 permit tcp from 212.101.4.244 to any 21 setup via em1
add 22302 permit tcp from 212.101.4.244 20 to any 1024-65535 setup via em1
add 22320 permit tcp from any 20 to 212.101.4.244 1024-65535 setup via em1
add 22340 permit tcp from any to 212.101.4.244 49152-65535 setup via em1

# RSYNC
add 22400 permit tcp from 212.101.4.244 to any rsync setup via em1
add 22410 permit tcp from any to me rsync setup via em1

# CVSUPD
add 22500 permit tcp from 212.101.4.244 to any cvsup setup via em1
add 22510 permit tcp from any to me cvsup setup via em1

#
# NO WINDOWS
#
add 30000 deny udp from any to any 137 via em1

# Internal Interface
add 62000  allow tcp from 212.101.0.0/24 to any 22,23,25 via em0 setup
add 62010  allow tcp from 212.101.1.0/24 to any 22,23,25 via em0 setup
add 62020  allow tcp from 212.101.4.0/24 to any 22,25 via em0 setup
add 63000  reset tcp from any to any 22,23,25,3306,111 via em0 setup
add 65000 allow ip from any to any via em0

# F*** off the rest
add 65500 deny log ip from any to any

If i load the ipfw rules i see:

 lock order reversal:
 1st 0xffffffff80839448 PFil hook read/write mutex (PFil hook read/write
mutex) _at_ /usr/src/sys/net/pfil.c:73
 2nd 0xffffffff80838440 in_multi_mtx (in_multi_mtx) _at_
/usr/src/sys/netinet/ip_output.c:302
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
witness_checkorder() at witness_checkorder+0x605
_mtx_lock_flags() at _mtx_lock_flags+0x75
ip_output() at ip_output+0x570
dummynet_send() at dummynet_send+0x13f
dummynet_io() at dummynet_io+0x322
ipfw_check_out() at ipfw_check_out+0x20a
pfil_run_hooks() at pfil_run_hooks+0xbc
ip_output() at ip_output+0x372
udp_send() at udp_send+0x350
sosend_dgram() at sosend_dgram+0x20f
kern_sendit() at kern_sendit+0x122
sendit() at sendit+0xdc
sendto() at sendto+0x4d
syscall() at syscall+0x1f6
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (133, FreeBSD ELF64, sendto), rip = 0x800bebc8c, rsp =
0x7fffffffe6c8, rbp = 0x551420 ---
Expensive timeout(9) function: 0xffffffff80469350(0xffffff004696e000)
0.530255650 s


FreeBSD is installed on ufs but I have a zfs pool for my data.
Filesystem       Size    Used   Avail Capacity  Mounted on
/dev/da0s1a      496M    356M    100M    78%    /
devfs            1.0K    1.0K      0B   100%    /dev
/dev/da0s1g      169G    4.0K    155G     0%    /disk1
/dev/da0s1f      3.9G    902K    3.6G     0%    /tmp
/dev/da0s1e       29G    3.9G     23G    15%    /usr
/dev/da0s1d       19G    4.0G     14G    22%    /var
pool             1.2T      0B    1.2T     0%    /usr/local/data
pool/cvsup       1.2T    5.1G    1.2T     0%    /usr/local/data/cvsup
pool/ftp         3.3T    2.2T    1.2T    65%    /usr/local/data/ftp
pool/portsnap    1.2T    591M    1.2T     0%    /usr/local/data/portsnap
pool/www         1.2T     78M    1.2T     0%    /usr/local/data/www

Any help?

Best regards,
Thomas
Received on Mon Dec 10 2007 - 14:55:37 UTC

This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:39:23 UTC