On Friday 09 April 2010 3:09:24 pm Jack Vogel wrote: > Someone else also pointed this out. I'm dubious about its claim. > This happens because there is an RX lock taken in rxeof, its held > thru the call into the stack, it then encounters another lock there > and hence this complaint. I've had the RX hold as it is for a long > while and would rather not have to give it up, can someone look > at it and advise? I've seen it happen with igb. I suspect it is a transitive lock order. That is, you probably never have the UDP lock acquired before an em/igb RX lock. However, if you have an em/igb adapter TX lock held when you acquire an em/igb RX lock in one place, and in if_start() you acquire the TX lock while the UDP lock is held, that can trigger the LOR. Specifically, those two paths would give you these two orders: TX -> RX UDP -> TX which implies the order UDP -> RX (lock order relationsips are transitive, just like a > b and b > c implies a > c). However, I haven't been able to track down what the raw orders are that might lead to this transitive order. Attilio added some sysctls to dump all the raw lock orders in one of the debug.witness sysctls. You can also try hardcoding the 'RX -> UDP' order using WITNESS_DEFINEORDER() before any of the em/igb RX/TX locks are acquired to see what different LOR is triggered. If that LOR looks valid then you can keep hardcoding valid orders until you find the invalid one. > On Fri, Apr 9, 2010 at 12:00 PM, Mike Tancsa <mike_at_sentex.net> wrote: > > > While testing an i5 box with HEAD checked out from this morning, bringing > > up the second NIC generated this LOR on the console > > > > em1: link state changed to UP > > lock order reversal: > > 1st 0xc5dc7c10 em1:rx(1) (em1:rx(1)) _at_ > > /usr/HEAD/src/sys/modules/em/../../dev/e1000/if_em.c:4089 > > 2nd 0xc0f7e88c udp (udp) _at_ /usr/HEAD/src/sys/netinet/udp_usrreq.c:454 > > KDB: stack backtrace: > > db_trace_self_wrapper(c0cb4d33,c5b72a70,c08e4d65,c08d50db,c0cb7d58,...) at > > db_trace_self_wrapper+0x26 > > kdb_backtrace(c08d50db,c0cb7d58,c5d31a98,c5d2cb60,c5b72acc,...) at > > kdb_backtrace+0x29 > > _witness_debugger(c0cb7d58,c0f7e88c,c0c9cf0b,c5d2cb60,c0cd04ca,...) at > > _witness_debugger+0x25 > > witness_checkorder(c0f7e88c,1,c0cd04ca,1c6,0,...) at > > witness_checkorder+0x839 > > _rw_rlock(c0f7e88c,c0cd04ca,1c6,c5d33088,c5e8be24,...) at _rw_rlock+0x9c > > udp_input(c67faa00,14,0,c5e8bd80,c0dfa860,...) at udp_input+0x246 > > ip_input(c67faa00,c5f2f380,c5b72be8,c07463b6,c0dfa860,...) at > > ip_input+0x606 > > netisr_dispatch_src(1,0,c67faa00,c5b72c20,c0954bc1,...) at > > netisr_dispatch_src+0xcd > > netisr_dispatch(1,c67faa00,c6018c00,c6018c00,c6852800,...) at > > netisr_dispatch+0x20 > > ether_demux(c6018c00,c67faa00,3,0,3,...) at ether_demux+0x1a1 > > ether_input(c6018c00,c67faa00,c11a0e17,ff9,64,...) at ether_input+0x365 > > em_rxeof(c5e8bd80,109,c6016180,0,c5b72cc8,...) at em_rxeof+0x13c > > em_msix_rx(c5dc7c00,c5b72cc8,c088eb14,c0e133c0,c60342b8,...) at > > em_msix_rx+0x25 > > intr_event_execute_handlers(c5d807f8,c6034280,c0cacd7e,533,c60342f0,...) at > > intr_event_execute_handlers+0x125 > > ithread_loop(c603b4a0,c5b72d38,c0cacaed,343,c5d807f8,...) at > > ithread_loop+0x9f > > fork_exit(c0877800,c603b4a0,c5b72d38) at fork_exit+0xb8 > > fork_trampoline() at fork_trampoline+0x8 > > --- trap 0, eip = 0, esp = 0xc5b72d70, ebp = 0 --- > > > > 0(i5b)# uname -a > > FreeBSD i5b.sentex.ca 9.0-CURRENT FreeBSD 9.0-CURRENT #2: Fri Apr 9 > > 11:56:25 EDT 2010 mdtancsa_at_ich10.sentex.ca:/usr/HEAD/obj/usr/HEAD/src/sys/GENERIC > > i386 > > 0(i5b)# > > > > em0_at_pci0:0:25:0: class=0x020000 card=0x34ec8086 chip=0x10ef8086 > > rev=0x05 hdr=0x00 > > vendor = 'Intel Corporation' > > class = network > > subclass = ethernet > > cap 01[c8] = powerspec 2 supports D0 D3 current D0 > > cap 05[d0] = MSI supports 1 message, 64 bit enabled with 1 message > > cap 13[e0] = PCI Advanced Features: FLR TP > > em1_at_pci0:3:0:0: class=0x020000 card=0x34ec8086 chip=0x10d38086 rev=0x00 > > hdr=0x00 > > vendor = 'Intel Corporation' > > device = 'Intel 82574L Gigabit Ethernet Controller (82574L)' > > class = network > > subclass = ethernet > > cap 01[c8] = powerspec 2 supports D0 D3 current D0 > > cap 05[d0] = MSI supports 1 message, 64 bit > > cap 10[e0] = PCI-Express 1 endpoint max data 128(256) link x1(x1) > > cap 11[a0] = MSI-X supports 5 messages in map 0x1c enabled > > > > > > 0(i5b)# vmstat -i > > interrupt total rate > > irq4: uart0 6156 3 > > irq8: rtc 224879 127 > > irq21: ehci0 2662 1 > > irq23: ehci1 3674 2 > > cpu0: timer 1754210 998 > > irq256: em0 10778 6 > > irq257: em1 331 0 > > irq258: em1 4 0 > > irq260: em1 4 0 > > irq261: em1 8 0 > > irq262: ahci0 69 0 > > cpu3: timer 1753938 998 > > cpu2: timer 1753932 998 > > cpu1: timer 1753886 998 > > Total 7264531 4134 > > 0(i5b)# > > > > CPU: Intel(R) Core(TM) i5 CPU 750 _at_ 2.67GHz (2666.65-MHz 686- class > > CPU) > > Origin = "GenuineIntel" Id = 0x106e5 Family = 6 Model = 1e Stepping = > > 5 > > > > Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE> > > > > Features2=0x98e3fd<SSE3,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,SSE4.1,SSE4.2,POPCNT> > > AMD Features=0x28100000<NX,RDTSCP,LM> > > AMD Features2=0x1<LAHF> > > TSC: P-state invariant > > real memory = 4294967296 (4096 MB) > > avail memory = 2577711104 (2458 MB) > > ACPI APIC Table: <INTEL S3420GPC> > > FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs > > > > > > > > > > -------------------------------------------------------------------- > > Mike Tancsa, tel +1 519 651 3400 > > Sentex Communications, mike_at_sentex.net > > Providing Internet since 1994 www.sentex.net > > Cambridge, Ontario Canada www.sentex.net/mike > > > > > _______________________________________________ > freebsd-current_at_freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-current > To unsubscribe, send any mail to "freebsd-current-unsubscribe_at_freebsd.org" > -- John BaldwinReceived on Mon Apr 12 2010 - 13:12:29 UTC
This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:40:02 UTC