Re: deadlock every 15-20 min

i found the reason of my unpredictable reboots.
it is in netgraph.ko module. if i don't let ppp(8) and mpd(8) load module
system works fine.
otherwise after a bit of time it hangs up with following error:

WARNING: attempt to net_add_domain(netgraph) after domainfinalize()
lock order reversal:
 1st 0xc2b8b090 inp (divinp) _at_
/usr/src/sys/modules/ipdivert/../../netinet/ip_divert.c:336
 2nd 0xc0a44db8 PFil hook read/write mutex (PFil hook read/write mutex) _at_
/usr/src/sys/net/pfil.c:73
KDB: stack backtrace:
kdb_backtrace(0,ffffffff,c0a04aa8,c0a04ee0,c09b1e84) at kdb_backtrace+0x29
witness_checkorder(c0a44db8,1,c0916282,49) at witness_checkorder+0x586
_rw_rlock(c0a44db8,c0916282,49) at _rw_rlock+0x54
pfil_run_hooks(c0a44da0,d4b00b3c,c29b2400,2,0) at pfil_run_hooks+0x2c
ip_output(c2aa9200,0,d4b00b08,22,0) at ip_output+0x627
div_send(c2b2c14c,0,c2aa9200,c2ab7950,0) at div_send+0x208
sosend(c2b2c14c,c2ab7950,d4b00be4,c2aa9200,0) at sosend+0x3e5
kern_sendit(c29bdbd0,3,d4b00c64,0,0) at kern_sendit+0x108
sendit(c29bdbd0,3,d4b00c64,0,bfbdebdf) at sendit+0x15f
sendto(c29bdbd0,d4b00d04,c2ad48d0,c,c29bdbd0) at sendto+0x4d
syscall(3b,3b,bfbf003b,2,4f) at syscall+0x27e
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (133, FreeBSD ELF32, sendto), eip = 0x2813069b, esp =
0xbfbdeafc, ebp = 0xbfbeeba8 ---



On 5/13/06, sekes <gexlie_at_gmail.com> wrote:
>
> i csuped and built new kernel this morning
> now every time i catch strong deadlock in the 15-20 minutes after booting
> up.
>
> i'm completely cutoff :-(
>
> this is the last produced message on the screen before the system hangs
> up.
>
> panic: rip_detach: inp == NULL
> cpuid = 0
> KDB: enter: panic
> [thread pid 11 tid 100005 ]
> Stopped at kdb_enter+0x2b: nop
> db>
>
> xnet# uname -a
> FreeBSD xnet.nnov.ru 7.0-CURRENT FreeBSD 7.0-CURRENT #0: Sat May 13
> 11:28:49 MSD 2006     root_at_xnet.nnov.ru:/usr/obj/usr/src/sys/GENERIC  i386
>
>
>
>
> Copyright (c) 1992-2006 The FreeBSD Project.
> Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
>         The Regents of the University of California. All rights reserved.
> FreeBSD 7.0-CURRENT #0: Sat May 13 11:28:49 MSD 2006
>     root_at_xnet.nnov.ru:/usr/obj/usr/src/sys/GENERIC
> WARNING: WITNESS option enabled, expect reduced performance.
> WARNING: MPSAFE network stack disabled, expect reduced performance.
> Timecounter "i8254" frequency 1193182 Hz quality 0
> CPU: Pentium II/Pentium II Xeon/Celeron (350.80-MHz 686-class CPU)
>   Origin = "GenuineIntel"  Id = 0x652  Stepping = 2
>
> Features=0x183f9ff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR>
> real memory  = 536858624 (511 MB)
> avail memory = 511541248 (487 MB)
> kbd1 at kbdmux0
> ath_hal: 0.9.16.16 (AR5210, AR5211, AR5212, RF5111, RF5112, RF2413,
> RF5413)
> ACPI disabled by blacklist.  Contact your BIOS vendor.
> cpu0 on motherboard
> pcib0: <Intel 82443BX (440 BX) host to PCI bridge> pcibus 0 on motherboard
>
> pir0: <PCI Interrupt Routing Table: 6 Entries> on motherboard
> pci0: <PCI bus> on pcib0
> agp0: <Intel 82443BX (440 BX) host to PCI bridge> on hostb0
> pcib1: <PCI-PCI bridge> at device 1.0 on pci0
> pci1: <PCI bus> on pcib1
> vgapci0: <VGA-compatible display> port 0x8a135330-0x8a13534f mem
> 0xdc000000-0xdfffffff at device 0.0 on pci1
> isab0: <PCI-ISA bridge> at device 4.0 on pci0
> isa0: <ISA bus> on isab0
> atapci0: <Intel PIIX4 UDMA33 controller> port
> 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xd800-0xd80f at device 4.1 on pci0
> ata0: <ATA channel 0> on atapci0
> ata1: <ATA channel 1> on atapci0
> uhci0: <Intel 82371AB/EB (PIIX4) USB controller> port 0xd400-0xd41f irq 5
> at device 4.2 on pci0
> uhci0: [GIANT-LOCKED]
> usb0: <Intel 82371AB/EB (PIIX4) USB controller> on uhci0
> usb0: USB revision 1.0
> uhub0: <Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb0
> uhub0: 2 ports with 2 removable, self powered
> piix0: <PIIX Timecounter> port 0xe800-0xe80f at device 4.3 on pci0
> Timecounter "PIIX" frequency 3579545 Hz quality 0
> rl0: <RealTek 8139 10/100BaseTX> port 0xd000-0xd0ff mem
> 0xdb800000-0xdb8000ff irq 12 at device 10.0 on pci0
> miibus0: <MII bus> on rl0
> rlphy0: <RealTek internal media interface> on miibus0
> rlphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
> rl0: Ethernet address: 00:80:48:31:0d:1f
> rl0: [GIANT-LOCKED]
> rl1: <RealTek 8139 10/100BaseTX> port 0xb800-0xb8ff mem
> 0xdb000000-0xdb0000ff irq 10 at device 11.0 on pci0
> miibus1: <MII bus> on rl1
> rlphy1: <RealTek internal media interface> on miibus1
> rlphy1:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
> rl1: Ethernet address: 00:80:48:25:36:9e
> rl1: [GIANT-LOCKED]
> rl2: <RealTek 8139 10/100BaseTX> port 0xb400-0xb4ff mem
> 0xda800000-0xda8000ff irq 11 at device 12.0 on pci0
> miibus2: <MII bus> on rl2
> rlphy2: <RealTek internal media interface> on miibus2
> rlphy2:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
> rl2: Ethernet address: 00:30:84:3a:e2:85
> rl2: [GIANT-LOCKED]
> pmtimer0 on isa0
> orm0: <ISA Option ROM> at iomem 0xc0000-0xc7fff pnpid ORM0000 on isa0
> atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
> atkbd0: <AT Keyboard> irq 1 on atkbdc0
> kbd0 at atkbd0
> atkbd0: [GIANT-LOCKED]
> fdc0: <Enhanced floppy controller> at port 0x3f0-0x3f5,0x3f7 irq 6 drq 2
> on isa0
> fdc0: [FAST]
> ppc0: <Parallel port> at port 0x378-0x37f irq 7 on isa0
> ppc0: SMC-like chipset (ECP/EPP/PS2/NIBBLE) in COMPATIBLE mode
> ppc0: FIFO with 16/16/9 bytes threshold
> ppbus0: <Parallel port bus> on ppc0
> plip0: <PLIP network interface> on ppbus0
> lpt0: <Printer> on ppbus0
> lpt0: Interrupt-driven port
> ppi0: <Parallel I/O> on ppbus0
> ppc0: [GIANT-LOCKED]
> sc0: <System console> at flags 0x100 on isa0
> sc0: EGA <16 virtual consoles, flags=0x300>
> sio0 at port 0x3f8-0x3ff irq 4 flags 0x10 on isa0
> sio0: type 16550A
> sio0: [FAST]
> sio1 at port 0x2f8-0x2ff irq 3 on isa0
> sio1: type 16550A
> sio1: [FAST]
> vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
> unknown: <PNP0401> can't assign resources (port)
> unknown: <PNP0501> can't assign resources (port)
> unknown: <PNP0501> can't assign resources (port)
> unknown: <PNP0700> can't assign resources (port)
> unknown: <PNP0c01> can't assign resources (memory)
> unknown: <PNP0303> can't assign resources (port)
> unknown: <PNP0c02> can't assign resources (port)
> Timecounter "TSC" frequency 350798189 Hz quality 800
> Timecounters tick every 1.000 msec
> ipfw2 (+ipv6) initialized, divert loadable, rule-based forwarding
> disabled, default to deny, logging disabled
> ad0: 9541MB <WDC WD100EB-00BHF0 15.15M15> at ata0-master UDMA33
> ad1: 39266MB <IC35L040AVVN07 0 VA2OAF0C> at ata0-slave UDMA33
> Trying to mount root from ufs:/dev/ad0s1a
> WARNING: / was not properly dismounted
> GEOM_ELI: Device ad0s1b.eli created.
> GEOM_ELI:     Cipher: AES
> GEOM_ELI: Key length: 256
> GEOM_ELI:     Crypto: software
> WARNING: /usr was not properly dismounted
> WARNING: /mnt/DISK1 was not properly dismounted
> pid 761 (ircd), uid 0: exited on signal 11 (core dumped)
> rl0: promiscuous mode enabled
> rl0: promiscuous mode disabled
> WARNING: attempt to net_add_domain(netgraph) after domainfinalize()
> lock order reversal:
>  1st 0xc2b8b090 inp (divinp) _at_
> /usr/src/sys/modules/ipdivert/../../netinet/ip_divert.c:336
>  2nd 0xc0a44db8 PFil hook read/write mutex (PFil hook read/write mutex) _at_
> /usr/src/sys/net/pfil.c:73
> KDB: stack backtrace:
> kdb_backtrace(0,ffffffff,c0a04aa8,c0a04ee0,c09b1e84) at kdb_backtrace+0x29
> witness_checkorder(c0a44db8,1,c0916282,49) at witness_checkorder+0x586
> _rw_rlock(c0a44db8,c0916282,49) at _rw_rlock+0x54
> pfil_run_hooks(c0a44da0,d4b00b3c,c29b2400,2,0) at pfil_run_hooks+0x2c
> ip_output(c2aa9200,0,d4b00b08,22,0) at ip_output+0x627
> div_send(c2b2c14c,0,c2aa9200,c2ab7950,0) at div_send+0x208
> sosend(c2b2c14c,c2ab7950,d4b00be4,c2aa9200,0) at sosend+0x3e5
> kern_sendit(c29bdbd0,3,d4b00c64,0,0) at kern_sendit+0x108
> sendit(c29bdbd0,3,d4b00c64,0,bfbdebdf) at sendit+0x15f
> sendto(c29bdbd0,d4b00d04,c2ad48d0,c,c29bdbd0) at sendto+0x4d
> syscall(3b,3b,bfbf003b,2,4f) at syscall+0x27e
> Xint0x80_syscall() at Xint0x80_syscall+0x1f
> --- syscall (133, FreeBSD ELF32, sendto), eip = 0x2813069b, esp =
> 0xbfbdeafc, ebp = 0xbfbeeba8 ---
> arp: 00:11:95:c4:ca:b0 attempts to modify permanent entry for 192.168.1.1on rl1
>