On 3/12/09, Bruce Simpson <bms_at_incunabulum.net> wrote: > Weongyo Jeong wrote: >> ... >> This is one I have got from "Paul B. Mahol" <onemda_at_gmail.com> >> yesterday and I think he might help you to get full backtrace: >> > Thanks for this. I really do need a full backtrace to work out what to > go on, however, > as it looks like the compiler is inlining static functions here. > > Is IGMPv2 in use on this network? The stack will default to IGMPv3, it > is possible that > the call to igmp_ifdetach() needs to be forced to happen before > in_purgemaddrs() causes > the IP stack's reference to the in_multi to be freed. > > Are any multicast applications in use? > > Sam is seeing a very different condition with 802.11 VAPs, even though > the symptoms are similar. > > I appreciate that it isn't always possible to test with all combinations > of network drivers > that people may be using, vlans vaps etc. and some of them do use parts > of the network stack > in different ways, i.e. stacking the use of otherwise refcounted > structures, etc. > > This looks like it is coming from USB2 and NDIS in some way. I am not at > all familiar with > those subsystems and don't use them, so help from others would be very > appreciated, > however I will try my best to help fix, the root cause may be more general. Here is one with usb2 and if_rum, (panic on detach happens only if inet address was asigned to wlan0: ifconfig wlan0 inet x.x.x.x) db:0:kdb.enter.unknown> run lockinfo db:1:lockinfo> show locks db:1:locks> show alllocks Process 733 (usbus4) thread 0xc43fc000 (100077) db:1:alllocks> show lockedvnods Locked vnodes db:0:kdb.enter.unknown> show pcpu cpuid = 0 curthread = 0xc43fc000: pid 733 "usbus4" curpcb = 0xc3bafd90 fpcurthread = none idlethread = 0xc3d09d20: pid 10 "idle: cpu0" APIC ID = 0 currentldt = 0x50 spin locks held: db:0:kdb.enter.unknown> bt Tracing pid 733 tid 100077 td 0xc43fc000 in_ifdetach(c3de9800,c3de9a30,32b,c3bafbc4,c4469ab1,...) at in_ifdetach+0x18d if_detach(c3de9800,0,c44a5d2c,237,c3de9800,...) at if_detach+0xfd rum_detach(c445b580,4,c06213f6,9e7,c04ce719,...) at rum_detach+0xab device_detach(c445b580,c42c9faa,c3f8d5e0,6,2,...) at device_detach+0x8c usb2_detach_device(c44bd000,ff,1,10,c061d235,...) at usb2_detach_device+0x16a uhub_explore(c4402000,0,c42c9267,d8,c4446d34,...) at uhub_explore+0x1ab usb2_bus_explore(c4446d34,0,c42d1f13,51,c068fdc0,...) at usb2_bus_explore+0xb9 usb2_process(c4446c70,c3bafd38,c061a98c,32d,c42967ec,...) at usb2_process+0xda fork_exit(c42bb3a0,c4446c70,c3bafd38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc3bafd70, ebp = 0 --- ugen4.2: <Ralink> at usbus4 rum0: <Ralink 802.11 bg WLAN, class 0/0, rev 2.00/0.01, addr 2> on usbus4 rum0: MAC/BBP RT2573 (rev 0x2573a), RF RT2528 rum0: need multicast update callback rum0: at uhub4, port 6, addr 2 (disconnected) Kernel page fault with the following non-sleepable locks held: exclusive sleep mutex if_addr_mtx (if_addr_mtx) r = 0 (0xc3de9a40) locked _at_ /usr/local/src/sys/netinet/in.c:1041 exclusive sleep mutex in_multi_mtx (in_multi_mtx) r = 0 (0xc07f9174) locked _at_ /usr/local/src/sys/netinet/in.c:1033 KDB: stack backtrace: db_trace_self_wrapper(c0621b7e,c3bafa64,c04e5995,c062e32e,409,...) at db_trace_self_wrapper+0x26 kdb_backtrace(c062e32e,409,ffffffff,c07cb19c,c3bafa9c,...) at kdb_backtrace+0x29 _witness_debugger(c0623edd,c3bafab0,4,1,0,...) at _witness_debugger+0x25 witness_warn(5,0,c06407ba,c3c8da90,c43fc000,...) at witness_warn+0x1fd trap(c3bafb3c) at trap+0x153 calltrap() at calltrap+0x6 --- trap 0xc, eip = 0xc055474d, esp = 0xc3bafb7c, ebp = 0xc3bafb9c --- in_ifdetach(c3de9800,c3de9a30,32b,c3bafbc4,c4469ab1,...) at in_ifdetach+0x18d if_detach(c3de9800,0,c44a5d2c,237,c3de9800,...) at if_detach+0xfd rum_detach(c445b580,4,c06213f6,9e7,c04ce719,...) at rum_detach+0xab device_detach(c445b580,c42c9faa,c3f8d5e0,6,2,...) at device_detach+0x8c usb2_detach_device(c44bd000,ff,1,10,c061d235,...) at usb2_detach_device+0x16a uhub_explore(c4402000,0,c42c9267,d8,c4446d34,...) at uhub_explore+0x1ab usb2_bus_explore(c4446d34,0,c42d1f13,51,c068fdc0,...) at usb2_bus_explore+0xb9 usb2_process(c4446c70,c3bafd38,c061a98c,32d,c42967ec,...) at usb2_process+0xda fork_exit(c42bb3a0,c4446c70,c3bafd38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc3bafd70, ebp = 0 --- Fatal trap 12: page fault while in kernel mode cpuid = 0; apic id = 00 fault virtual address = 0x0 fault code = supervisor write, page not present instruction pointer = 0x20:0xc055474d stack pointer = 0x28:0xc3bafb7c frame pointer = 0x28:0xc3bafb9c code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, def32 1, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 733 (usbus4) exclusive sleep mutex if_addr_mtx (if_addr_mtx) r = 0 (0xc3de9a40) locked _at_ /usr/local/src/sys/netinet/in.c:1041 exclusive sleep mutex in_multi_mtx (in_multi_mtx) r = 0 (0xc07f9174) locked _at_ /usr/local/src/sys/netinet/in.c:1033 exclusive sleep mutex Giant (Giant) r = 0 (0xc068b810) locked _at_ /usr/local/src/sys/modules/usb/usb/../../../dev/usb/controller/usb_controller.c:216 exclusive sx 123456789ABCDEF - USB config SX lock (123456789ABCDEF - USB config SX lock) r = 0 (0xc44bd03c) locked _at_ /usr/local/src/sys/modules/usb/usb/../../../dev/usb/usb_device.c:941 exclusive sleep mutex if_addr_mtx (if_addr_mtx) r = 0 (0xc3de9a40) locked _at_ /usr/local/src/sys/netinet/in.c:1041 exclusive sleep mutex in_multi_mtx (in_multi_mtx) r = 0 (0xc07f9174) locked _at_ /usr/local/src/sys/netinet/in.c:1033 exclusive sleep mutex Giant (Giant) r = 0 (0xc068b810) locked _at_ /usr/local/src/sys/modules/usb/usb/../../../dev/usb/controller/usb_controller.c:216 exclusive sx 123456789ABCDEF - USB config SX lock (123456789ABCDEF - USB config SX lock) r = 0 (0xc44bd03c) locked _at_ /usr/local/src/sys/modules/usb/usb/../../../dev/usb/usb_device.c:941 (gdb) l *in_ifdetach+0x18d 0xc055474d is in in_ifdetach (/usr/local/src/sys/netinet/in.c:1046). 1041 IF_ADDR_LOCK(ifp); 1042 TAILQ_FOREACH(ifma, &ifp->if_multiaddrs, ifma_link) { 1043 if (ifma->ifma_addr->sa_family != AF_INET) 1044 continue; 1045 inm = (struct in_multi *)ifma->ifma_protospec; 1046 LIST_INSERT_HEAD(&purgeinms, inm, inm_link); 1047 } 1048 IF_ADDR_UNLOCK(ifp); 1049 1050 LIST_FOREACH_SAFE(inm, &purgeinms, inm_link, tinm) { (gdb) l *if_detach+0xfd 0xc054285d is in if_detach (/usr/local/src/sys/net/if.c:847). 842 * routes are expected to be removed by the IPv6-specific kernel API. 843 * Otherwise, the kernel will detect some inconsistency and bark it. 844 */ 845 in6_ifdetach(ifp); 846 #endif 847 if_purgemaddrs(ifp); 848 849 /* 850 * Remove link ifaddr pointer and maybe decrement if_index. 851 * Clean up all addresses. (gdb) l *rum_detach+0xab 0x27fb is in rum_detach (/usr/local/src/sys/modules/usb/rum/../../../dev/usb/wlan/if_rum.c:573). 568 569 if (ifp) { 570 ic = ifp->if_l2com; 571 bpfdetach(ifp); 572 ieee80211_ifdetach(ic); 573 if_free(ifp); 574 } 575 cv_destroy(&sc->sc_cmd_cv); 576 mtx_destroy(&sc->sc_mtx); 577 Here is same crash, but via kgdb: (kgdb) bt #0 doadump () at pcpu.h:246 #1 0xc04a744e in boot (howto=260) at /usr/local/src/sys/kern/kern_shutdown.c:420 #2 0xc04a7722 in panic (fmt=Variable "fmt" is not available. ) at /usr/local/src/sys/kern/kern_shutdown.c:576 #3 0xc05f8ba3 in trap_fatal (frame=0xc3bdfb3c, eva=0) at /usr/local/src/sys/i386/i386/trap.c:926 #4 0xc05f9441 in trap (frame=0xc3bdfb3c) at /usr/local/src/sys/i386/i386/trap.c:318 #5 0xc05dfd9b in calltrap () at /usr/local/src/sys/i386/i386/exception.s:165 #6 0xc055474d in in_ifdetach (ifp=0xc3fc9c00) at /usr/local/src/sys/netinet/in.c:1017 #7 0xc054285d in if_detach (ifp=0xc3fc9c00) at /usr/local/src/sys/net/if.c:835 #8 0xc45087fb in rum_detach (self=0xc44e6d80) at /usr/local/src/sys/modules/usb/rum/../../../dev/usb/wlan/if_rum.c:572 #9 0xc04ceb5c in device_detach (dev=0xc44e6d80) at device_if.h:212 #10 0xc434998a in usb2_detach_device (udev=0xc44b2000, iface_index=32 ' ', free_subdev=1 '\001') at /usr/local/src/sys/modules/usb/usb/../../../dev/usb/usb_device.c:901 #11 0xc4351d2b in uhub_explore (udev=0xc4156000) at /usr/local/src/sys/modules/usb/usb/../../../dev/usb/usb_hub.c:301 #12 0xc4342fb9 in usb2_bus_explore (pm=0xc43fbd34) at /usr/local/src/sys/modules/usb/usb/../../../dev/usb/controller/usb_controller.c:227 #13 0xc435447a in usb2_process (arg=0xc43fbc70) at /usr/local/src/sys/modules/usb/usb/../../../dev/usb/usb_process.c:139 #14 0xc0484458 in fork_exit (callout=0xc43543a0 <usb2_process>, arg=0xc43fbc70, frame=0xc3bdfd38) at /usr/local/src/sys/kern/kern_fork.c:821 #15 0xc05dfe10 in fork_trampoline () at /usr/local/src/sys/i386/i386/exception.s:270 (kgdb) frame 6 #6 0xc055474d in in_ifdetach (ifp=0xc3fc9c00) at /usr/local/src/sys/netinet/in.c:1017 1017 } (kgdb) frame 5 #5 0xc05dfd9b in calltrap () at /usr/local/src/sys/i386/i386/exception.s:165 165 call trap Current language: auto; currently asm (kgdb) frame 7 #7 0xc054285d in if_detach (ifp=0xc3fc9c00) at /usr/local/src/sys/net/if.c:835 835 in_ifdetach(ifp); Current language: auto; currently c -- PaulReceived on Thu Mar 12 2009 - 18:37:35 UTC
This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:39:43 UTC