On Mon, Dec 10, 2007 at 04:36:23PM +0100, Thomas Vogt wrote: > Hi, > > I kindly ask for advice for debugging a problem with 7.0-Beta 4 (amd64). > > The machine freezes if dummynet rules are enabled on a gigabit port. The > system runs fine on fast ethernet or with dummynet rules disabled. I > can't trigger the bug if my upstream port is limited to 100mbit. The > machine suddenly freezes after 3-4 minutes if dummynet rules are loaded. > > My Kernel already includes some debug options but I'm not able to > produce a panic. > > Kernel: > options IPFIREWALL > options IPFIREWALL_VERBOSE > options IPFIREWALL_VERBOSE_LIMIT=100 > options IPFIREWALL_DEFAULT_TO_ACCEPT > options DUMMYNET > options ZERO_COPY_SOCKETS > options DEVICE_POLLING > options HZ=1000 > > options INVARIANTS > options INVARIANT_SUPPORT > options WITNESS > options DEBUG_VFS_LOCKS > options DIAGNOSTIC > options KDB > options KDB_UNATTENDED > options DDB > options GDB > options KDB_TRACE > options SOCKBUF_DEBUG > options BREAK_TO_DEBUGGER > > dmesg: > Copyright (c) 1992-2007 The FreeBSD Project. > Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 > The Regents of the University of California. All rights reserved. > FreeBSD is a registered trademark of The FreeBSD Foundation. > FreeBSD 7.0-BETA4 #4: Fri Dec 7 12:45:37 UTC 2007 > root_at_lisa.mlan.solnet.ch:/usr/obj/usr/src/sys/SOLNETDEBUG > WARNING: WITNESS option enabled, expect reduced performance. > WARNING: DIAGNOSTIC option enabled, expect reduced performance. > Timecounter "i8254" frequency 1193182 Hz quality 0 > CPU: Intel(R) Xeon(R) CPU 5110 _at_ 1.60GHz (1739.55-MHz > K8-class CPU) > Origin = "GenuineIntel" Id = 0x6f6 Stepping = 6 > > Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE> > Features2=0x4e33d<SSE3,RSVD2,MON,DS_CPL,VMX,TM2,SSSE3,CX16,xTPR,PDCM,DCA> > AMD Features=0x20100800<SYSCALL,NX,LM> > AMD Features2=0x1<LAHF> > Cores per package: 2 > usable memory = 4258033664 (4060 MB) > avail memory = 4071751680 (3883 MB) > ACPI APIC Table: <INTEL S5000XVN> > FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs > cpu0 (BSP): APIC ID: 0 > cpu1 (AP): APIC ID: 1 > WITNESS: spin lock intrcnt not in order list > ioapic0 <Version 2.0> irqs 0-23 on motherboard > ioapic1 <Version 2.0> irqs 24-47 on motherboard > lapic0: Forcing LINT1 to edge trigger > kbd1 at kbdmux0 > acpi0: <INTEL S5000XVN> on motherboard > acpi0: [ITHREAD] > acpi0: Power Button (fixed) > acpi0: reservation of 0, a0000 (3) failed > Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000 > acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0 > acpi_hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on > acpi0 > Timecounter "HPET" frequency 14318180 Hz quality 900 > cpu0: <ACPI CPU> on acpi0 > device_attach: acpi_perf0 attach returned 6 > device_attach: acpi_perf0 attach returned 6 > p4tcc0: <CPU Frequency Thermal Control> on cpu0 > cpu1: <ACPI CPU> on acpi0 > device_attach: acpi_perf1 attach returned 6 > device_attach: acpi_perf1 attach returned 6 > p4tcc1: <CPU Frequency Thermal Control> on cpu1 > acpi_button0: <Sleep Button> on acpi0 > acpi_button1: <Power Button> on acpi0 > pcib0: <ACPI Host-PCI bridge> port 0xca2,0xca3,0xcf8-0xcff on acpi0 > pci0: <ACPI PCI bus> on pcib0 > pcib1: <ACPI PCI-PCI bridge> at device 2.0 on pci0 > pci1: <ACPI PCI bus> on pcib1 > pcib2: <ACPI PCI-PCI bridge> irq 16 at device 0.0 on pci1 > pci2: <ACPI PCI bus> on pcib2 > pcib3: <ACPI PCI-PCI bridge> irq 16 at device 0.0 on pci2 > pci3: <ACPI PCI bus> on pcib3 > pcib4: <PCI-PCI bridge> at device 0.0 on pci3 > pci4: <PCI bus> on pcib4 > arcmsr0: <Areca SATA Host Adapter RAID Controller (RAID6 capable) > > mem 0xc3d00000-0xc3d00fff,0xc3000000-0xc33fffff irq 18 at device 14.0 > on pci4 > ARECA RAID ADAPTER0: Driver Version 1.20.00.14 2007-2-05 > ARECA RAID ADAPTER0: FIRMWARE VERSION V1.43 2007-4-17 > arcmsr0: [ITHREAD] > pcib5: <PCI-PCI bridge> at device 0.2 on pci3 > pci5: <PCI bus> on pcib5 > pcib6: <ACPI PCI-PCI bridge> irq 18 at device 2.0 on pci2 > pci6: <ACPI PCI bus> on pcib6 > em0: <Intel(R) PRO/1000 Network Connection Version - 6.7.3> port > 0x2020-0x203f mem 0xc3c20000-0xc3c3ffff,0xc3800000-0xc3bfffff irq 18 at > device 0.0 on pci6 > em0: Using MSI interrupt > em0: Ethernet address: 00:15:17:30:94:30 > em0: [FILTER] > em1: <Intel(R) PRO/1000 Network Connection Version - 6.7.3> port > 0x2000-0x201f mem 0xc3c00000-0xc3c1ffff,0xc3400000-0xc37fffff irq 19 at > device 0.1 on pci6 > em1: Using MSI interrupt > em1: Ethernet address: 00:15:17:30:94:31 > em1: [FILTER] > pcib7: <ACPI PCI-PCI bridge> at device 0.3 on pci1 > pci7: <ACPI PCI bus> on pcib7 > pcib8: <PCI-PCI bridge> at device 3.0 on pci0 > pci8: <PCI bus> on pcib8 > pcib9: <ACPI PCI-PCI bridge> at device 4.0 on pci0 > pci9: <ACPI PCI bus> on pcib9 > vgapci0: <VGA-compatible display> port 0x1000-0x107f mem > 0xc2000000-0xc2ffffff,0xb0000000-0xbfffffff,0xc0000000-0xc1ffffff irq 16 > at device 0.0 on pci9 > pcib10: <ACPI PCI-PCI bridge> at device 5.0 on pci0 > pci10: <ACPI PCI bus> on pcib10 > pcib11: <ACPI PCI-PCI bridge> at device 6.0 on pci0 > pci11: <ACPI PCI bus> on pcib11 > pcib12: <PCI-PCI bridge> at device 7.0 on pci0 > pci12: <PCI bus> on pcib12 > pci0: <base peripheral> at device 8.0 (no driver attached) > pci0: <multimedia> at device 27.0 (no driver attached) > pcib13: <ACPI PCI-PCI bridge> irq 16 at device 28.0 on pci0 > pci13: <ACPI PCI bus> on pcib13 > uhci0: <Intel 631XESB/632XESB/3100 USB controller USB-1> port > 0x3080-0x309f irq 23 at device 29.0 on pci0 > uhci0: [GIANT-LOCKED] > uhci0: [ITHREAD] > usb0: <Intel 631XESB/632XESB/3100 USB controller USB-1> on uhci0 > usb0: USB revision 1.0 > uhub0: <Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb0 > uhub0: 2 ports with 2 removable, self powered > uhci1: <Intel 631XESB/632XESB/3100 USB controller USB-2> port > 0x3060-0x307f irq 22 at device 29.1 on pci0 > uhci1: [GIANT-LOCKED] > uhci1: [ITHREAD] > usb1: <Intel 631XESB/632XESB/3100 USB controller USB-2> on uhci1 > usb1: USB revision 1.0 > uhub1: <Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb1 > uhub1: 2 ports with 2 removable, self powered > uhci2: <Intel 631XESB/632XESB/3100 USB controller USB-3> port > 0x3040-0x305f irq 23 at device 29.2 on pci0 > uhci2: [GIANT-LOCKED] > uhci2: [ITHREAD] > usb2: <Intel 631XESB/632XESB/3100 USB controller USB-3> on uhci2 > usb2: USB revision 1.0 > uhub2: <Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb2 > uhub2: 2 ports with 2 removable, self powered > uhci3: <Intel 631XESB/632XESB/3100 USB controller USB-4> port > 0x3020-0x303f irq 22 at device 29.3 on pci0 > uhci3: [GIANT-LOCKED] > uhci3: [ITHREAD] > usb3: <Intel 631XESB/632XESB/3100 USB controller USB-4> on uhci3 > usb3: USB revision 1.0 > uhub3: <Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb3 > uhub3: 2 ports with 2 removable, self powered > ehci0: <Intel 63XXESB USB 2.0 controller> mem 0xc3f04400-0xc3f047ff irq > 23 at device 29.7 on pci0 > ehci0: [GIANT-LOCKED] > ehci0: [ITHREAD] > usb4: EHCI version 1.0 > usb4: companion controllers, 2 ports each: usb0 usb1 usb2 usb3 > usb4: <Intel 63XXESB USB 2.0 controller> on ehci0 > usb4: USB revision 2.0 > uhub4: <Intel EHCI root hub, class 9/0, rev 2.00/1.00, addr 1> on usb4 > uhub4: 8 ports with 8 removable, self powered > pcib14: <ACPI PCI-PCI bridge> at device 30.0 on pci0 > pci14: <ACPI PCI bus> on pcib14 > isab0: <PCI-ISA bridge> at device 31.0 on pci0 > isa0: <ISA bus> on isab0 > atapci0: <Intel 63XXESB2 UDMA100 controller> port > 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x30b0-0x30bf irq 20 at device 31.1 > on pci0 > ata0: <ATA channel 0> on atapci0 > ata0: [ITHREAD] > ata1: <ATA channel 1> on atapci0 > ata1: [ITHREAD] > atapci1: <Intel 63XXESB2 SATA300 controller> port > 0x30c8-0x30cf,0x30e4-0x30e7,0x30c0-0x30c7,0x30e0-0x30e3,0x30a0-0x30af > mem 0xc3f04000-0xc3f043ff irq 20 at device 31.2 on pci0 > atapci1: [ITHREAD] > ata2: <ATA channel 0> on atapci1 > ata2: [ITHREAD] > ata3: <ATA channel 1> on atapci1 > ata3: [ITHREAD] > pci0: <serial bus, SMBus> at device 31.3 (no driver attached) > atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0 > atkbd0: <AT Keyboard> irq 1 on atkbdc0 > kbd0 at atkbd0 > atkbd0: [GIANT-LOCKED] > atkbd0: [ITHREAD] > sio0: configured irq 4 not in bitmap of probed irqs 0 > sio0: port may not be enabled > sio0: configured irq 4 not in bitmap of probed irqs 0 > sio0: port may not be enabled > sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 flags 0x10 on > acpi0 > sio0: type 16550A, console > sio0: [FILTER] > sio1: configured irq 3 not in bitmap of probed irqs 0 > sio1: port may not be enabled > sio1: configured irq 3 not in bitmap of probed irqs 0 > sio1: port may not be enabled > sio1: <16550A-compatible COM port> port 0x2f8-0x2ff irq 3 on acpi0 > sio1: type 16550A > sio1: [FILTER] > orm0: <ISA Option ROMs> at iomem > 0xcd000-0xcdfff,0xce000-0xcefff,0xcf000-0xcffff on isa0 > ppc0: cannot reserve I/O port range > sc0: <System console> at flags 0x100 on isa0 > sc0: VGA <16 virtual consoles, flags=0x300> > vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 > Timecounters tick every 1.000 msec > ipfw2 (+ipv6) initialized, divert loadable, rule-based forwarding > disabled, default to accept, logging limited to 100 packets/entry by default > Waiting 5 seconds for SCSI devices to settle > acd0: DVDROM <HL-DT-STDVD-ROM GDR8161B/0100> at ata0-master UDMA33 > da0 at arcmsr0 bus 0 target 0 lun 0 > da0: <Areca ARC-1230-VOL#00 R001> Fixed Direct Access SCSI-5 device > da0: 166.666MB/s transfers (83.333MHz DT, offset 32, 16bit) > da0: 238418MB (488281088 512 byte sectors: 255H 63S/T 30394C) > da1 at arcmsr0 bus 0 target 0 lun 1 > da1: <Seagate ST3750640NS R001> Fixed Direct Access SCSI-5 device > da1: 166.666MB/s transfers (83.333MHz DT, offset 32, 16bit) > da1: 715404MB (1465149168 512 byte sectors: 255H 63S/T 91201C) > da2 at arcmsr0 bus 0 target 0 lun 2 > da2: <Seagate ST3750640NS R001> Fixed Direct Access SCSI-5 device > da2: 166.666MB/s transfers (83.333MHz DT, offset 32, 16bit) > da2: 715404MB (1465149168 512 byte sectors: 255H 63S/T 91201C) > da3 at arcmsr0 bus 0 target 0 lun 3 > da3: <Seagate ST3750640NS R001> Fixed Direct Access SCSI-5 device > da3: 166.666MB/s transfers (83.333MHz DT, offset 32, 16bit) > da3: 715404MB (1465149168 512 byte sectors: 255H 63S/T 91201C) > da4 at arcmsr0 bus 0 target 0 lun 4 > da4: <Seagate ST3750640NS R001> Fixed Direct Access SCSI-5 device > da4: 166.666MB/s transfers (83.333MHz DT, offset 32, 16bit) > da4: 715404MB (1465149168 512 byte sectors: 255H 63S/T 91201C) > da5 at arcmsr0 bus 0 target 0 lun 5 > da5: <Seagate ST3750640NS R001> Fixed Direct Access SCSI-5 device > da5: 166.666MB/s transfers (83.333MHz DT, offset 32, 16bit) > da5: 715404MB (1465149168 512 byte sectors: 255H 63S/T 91201C) > da6 at arcmsr0 bus 0 target 0 lun 6 > da6: <Seagate ST3750640NS R001> Fixed Direct Access SCSI-5 device > da6: 166.666MB/s transfers (83.333MHz DT, offset 32, 16bit) > da6: 715404MB (1465149168 512 byte sectors: 255H 63S/T 91201C) > da7 at arcmsr0 bus 0 target 0 lun 7 > da7: <Seagate ST3750640NS R001> Fixed Direct Access SCSI-5 device > da7: 166.666MB/s transfers (83.333MHz DT, offset 32, 16bit) > da7: 715404MB (1465149168 512 byte sectors: 255H 63S/T 91201C) > lapic1: Forcing LINT1 to edge trigger > SMP: AP CPU #1 Launched! > WARNING: WITNESS option enabled, expect reduced performance. > WARNING: DIAGNOSTIC option enabled, expect reduced performance. > Trying to mount root from ufs:/dev/da0s1a > WARNING: ZFS is considered to be an experimental feature in FreeBSD. > ZFS filesystem version 6 > ZFS storage pool version 6 > em1: Hardware Initialization Failed > em1: Unable to initialize the hardware > Expensive timeout(9) function: 0xffffffff80258fe0(0xffffff00012fc000) > 0.008566172 s > em1: link state changed to UP > em1: link state changed to DOWN > em1: link state changed to UP > Expensive timeout(9) function: 0xffffffff80507ae0(0) 0.105003251 s > Expensive timeout(9) function: 0xffffffff80397260(0) 0.372032111 s > Expensive timeout(9) function: 0xffffffff80469350(0xffffff0069b5f888) > 0.403952336 s > > > my ipfw rules: > pipe 2 config bw 600Mbit/s queue 50KBytes > queue 201 config pipe 2 weight 1 mask dst-ip 0x0000000f queue 50 > queue 210 config pipe 2 weight 50 mask dst-ip 0x0000000f queue 50 > > > add 100 permit ip from any to any via lo0 > add 200 deny ip from any to 127.0.0.0/8 > > # Speedup Established > add 5001 skipto 8000 ip from 82.220.2.0/24 to 212.101.0.0/19 via em1 > add 6000 queue 210 ip from any to 212.101.0.0/19 out xmit em1 > add 6001 skipto 8000 ip from any to 212.101.0.0/19 out xmit em1 > add 6002 queue 210 ip from any to 212.41.65.0/18 out xmit em1 > add 6003 skipto 8000 ip from any to 212.41.65.0/18 out xmit em1 > add 6004 queue 210 ip from any to 82.220.0.0/18 out xmit em1 > add 6005 skipto 8000 ip from any to 82.220.0.0/18 out xmit em1 > add 6100 queue 201 ip from any to any out xmit em1 > > add 8000 permit tcp from any to any via em1 established > > # ICMP > add 11700 permit icmp from any to any via em1 icmptype 0,3,4,8,11,12 > > # Traceroute > add 1800 allow udp from any 30000-60000 to me 30000-60000 via em0 > > > # SSH (external connections are allowed for portsnap maintainer) > add 11900 allow ip from me to any ssh setup > add 11901 allow ip from any to me ssh setup > > # HTTP OUT > add 11910 permit tcp from me to any 80 setup via em1 > > # SMTP > add 12100 permit tcp from me to 212.101.0.0/19 25 setup via em1 > add 12110 allow tcp from me to any dst-port smtp via em1 setup > > # DNS UDP SolNet > add 21100 permit udp from 212.101.0.10 domain to me recv em1 > add 21101 permit udp from me to 212.101.0.10 domain out xmit em1 > add 21110 permit udp from 212.101.4.253 domain to me recv em1 > add 21111 permit udp from me to 212.101.4.253 domain out xmit em1 > add 21120 permit udp from 212.101.4.251 domain to me recv em1 > add 21121 permit udp from me to 212.101.4.251 domain out xmit em1 > > # NTP > add 21440 permit udp from me ntp to 212.101.0.0/19 ntp out xmit em1 > add 21445 permit udp from 224.0.1.1 ntp to 212.101.0.0/19 ntp out xmit em1 > add 21450 permit udp from 212.101.0.0/19 ntp to me ntp recv em1 > add 21460 permit udp from 212.101.0.0/19 ntp to 224.0.1.1 ntp recv em1 > add 21470 permit udp from me ntp to 224.0.1.1 ntp out xmit em1 > > > # WEB > add 22250 permit tcp from any to 212.101.4.241 80 setup via em1 limit > src-addr 4 > add 22251 permit tcp from any to 212.101.4.241 443 setup via em1 limit > src-addr 4 > add 22253 permit tcp from 212.101.4.241 to any 1024-65535 setup via em1 > > > # FTP > add 22300 permit tcp from any to 212.101.4.244 21 setup via em1 limit > src-addr 4 > add 22301 permit tcp from 212.101.4.244 to any 21 setup via em1 > add 22302 permit tcp from 212.101.4.244 20 to any 1024-65535 setup via em1 > add 22320 permit tcp from any 20 to 212.101.4.244 1024-65535 setup via em1 > add 22340 permit tcp from any to 212.101.4.244 49152-65535 setup via em1 > > # RSYNC > add 22400 permit tcp from 212.101.4.244 to any rsync setup via em1 > add 22410 permit tcp from any to me rsync setup via em1 > > # CVSUPD > add 22500 permit tcp from 212.101.4.244 to any cvsup setup via em1 > add 22510 permit tcp from any to me cvsup setup via em1 > > # > # NO WINDOWS > # > add 30000 deny udp from any to any 137 via em1 > > # Internal Interface > add 62000 allow tcp from 212.101.0.0/24 to any 22,23,25 via em0 setup > add 62010 allow tcp from 212.101.1.0/24 to any 22,23,25 via em0 setup > add 62020 allow tcp from 212.101.4.0/24 to any 22,25 via em0 setup > add 63000 reset tcp from any to any 22,23,25,3306,111 via em0 setup > add 65000 allow ip from any to any via em0 > > # F*** off the rest > add 65500 deny log ip from any to any > > If i load the ipfw rules i see: > > lock order reversal: > 1st 0xffffffff80839448 PFil hook read/write mutex (PFil hook read/write > mutex) _at_ /usr/src/sys/net/pfil.c:73 > 2nd 0xffffffff80838440 in_multi_mtx (in_multi_mtx) _at_ > /usr/src/sys/netinet/ip_output.c:302 > KDB: stack backtrace: > db_trace_self_wrapper() at db_trace_self_wrapper+0x2a > witness_checkorder() at witness_checkorder+0x605 > _mtx_lock_flags() at _mtx_lock_flags+0x75 > ip_output() at ip_output+0x570 > dummynet_send() at dummynet_send+0x13f > dummynet_io() at dummynet_io+0x322 > ipfw_check_out() at ipfw_check_out+0x20a > pfil_run_hooks() at pfil_run_hooks+0xbc > ip_output() at ip_output+0x372 > udp_send() at udp_send+0x350 > sosend_dgram() at sosend_dgram+0x20f > kern_sendit() at kern_sendit+0x122 > sendit() at sendit+0xdc > sendto() at sendto+0x4d > syscall() at syscall+0x1f6 > Xfast_syscall() at Xfast_syscall+0xab > --- syscall (133, FreeBSD ELF64, sendto), rip = 0x800bebc8c, rsp = > 0x7fffffffe6c8, rbp = 0x551420 --- > Expensive timeout(9) function: 0xffffffff80469350(0xffffff004696e000) > 0.530255650 s > > > FreeBSD is installed on ufs but I have a zfs pool for my data. > Filesystem Size Used Avail Capacity Mounted on > /dev/da0s1a 496M 356M 100M 78% / > devfs 1.0K 1.0K 0B 100% /dev > /dev/da0s1g 169G 4.0K 155G 0% /disk1 > /dev/da0s1f 3.9G 902K 3.6G 0% /tmp > /dev/da0s1e 29G 3.9G 23G 15% /usr > /dev/da0s1d 19G 4.0G 14G 22% /var > pool 1.2T 0B 1.2T 0% /usr/local/data > pool/cvsup 1.2T 5.1G 1.2T 0% /usr/local/data/cvsup > pool/ftp 3.3T 2.2T 1.2T 65% /usr/local/data/ftp > pool/portsnap 1.2T 591M 1.2T 0% /usr/local/data/portsnap > pool/www 1.2T 78M 1.2T 0% /usr/local/data/www > > Any help? > > Best regards, > Thomas > _______________________________________________ > freebsd-current_at_freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-current > To unsubscribe, send any mail to "freebsd-current-unsubscribe_at_freebsd.org" Please try this patch: http://www.freebsd.org/cgi/query-pr.cgi?prp=113548-3-diff and report does it help or not. -- Oleg. ================================================================ === Oleg Bulyzhin -- OBUL-RIPN -- OBUL-RIPE -- oleg_at_rinet.ru === ================================================================Received on Mon Dec 10 2007 - 18:10:39 UTC
This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:39:23 UTC