Re: LOR route vr0

John Baldwin wrote:
> On Thursday 01 September 2005 01:22 pm, Don Lewis wrote:
> 
>>On  1 Sep, Fredrik Lindberg wrote:
>>
>>>Don Lewis wrote:
>>>
>>>>On 27 Aug, M. Warner Losh wrote:
>>>>
>>>>>In message: <20050828025721.X43518_at_fledge.watson.org>
>>>>>
>>>>>           Robert Watson <rwatson_at_FreeBSD.org> writes:
>>>>>: On Sat, 27 Aug 2005, M. Warner Losh wrote:
>>>>>: > : You need to add an entry to subr_witness.c creating a graph edge
>>>>>: > : between the softc lock and the routing lock.  An example of an
>>>>>: > : entry in subr_witness.c:
>>>>>: > :
>>>>>: > :          /*
>>>>>: > :           * TCP/IP
>>>>>: > :           */
>>>>>: > :          { "tcp", &lock_class_mtx_sleep },
>>>>>: > :          { "tcpinp", &lock_class_mtx_sleep },
>>>>>: > :          { "so_snd", &lock_class_mtx_sleep },
>>>>>: > :          { NULL, NULL },
>>>>>: > :
>>>>>: > : Note that sets of ordered entries are terminated with a
>>>>>: > : double-null.  This declares that locks of type "tcp" preceed
>>>>>: > : "tcpinp" which preceed "so_snd".
>>>>>: >
>>>>>: > So you have to have locks of type tcp BEFORE you take out tcpinp
>>>>>: > type locks?
>>>>>:
>>>>>: Correct.  'tcp' reflects the global TCP state tables (pcbinfo) locks,
>>>>>: and 'tcpinp' is for individual PCBs.  If you acquire first a tcpinp
>>>>>: and then tcp, the above settings should cause WITNESS to generate a
>>>>>: lock order warning.  Likewise, both tcp and tcpinp preceed so_snd, so
>>>>>: if you acquire a protocol lock after a socket lock, it will get
>>>>>: unhappy.  WITNESS handles transitive relationships, so it gets
>>>>>: connected up to the rest of the lock graph, explicit and implicit, so
>>>>>: indirect violations of orders are fully handled.
>>>>>
>>>>>OK.  I've been seeing similar LORs in ed, sn, iwi (ed is my locked
>>>>>version of ed, not in tree GIANT locked ed).
>>>>
>>>>Just as a datapoint, I've got fxp interfaces on all my machines running
>>>>-CURRENT and I'm not seeing the problem here.
>>>
>>>I'm seeing both the rentry and the tcpinp LORs on my fxp interface
>>>on a machine running a few days old -current (Aug 25).
>>>
>>>lock order reversal
>>>1st 0xc1e30d38 inp (tcpinp) _at_ /usr/src/sys/netinet/tcp_input.c:742
>>>2nd 0xc1b74018 fxp0 (network driver) _at_/usr/src/sys/dev/fxp/if_fxp.c:1172
>>>
>>>lock order reversal
>>>1st 0xc1e06bb8 rtentry (rtentry) _at_ /usr/src/sys/net/route.c:1269
>>>2nd 0xc1b74018 fxp0 (network driver) _at_/usr/src/sys/dev/fxp/if_fxp.c:1172
>>>
>>>As for their backtraces they are almost identical to the
>>>once already posted.
>>
>>Are you using any applications that use multicast?  Can you break into
>>DDB and capture the output of "show witness"?
>
> 
> Also, are you using DEVICE_POLLING?
> 

No, I'm not using any multicast applications, however the LORs are 
always triggerd by dhclient at boot, but only at the first
DHCPDISCOVER. Initiating a a new discovery does not trigger a LOR.
I had DEVICE_POLLLING in my kernel but never used it, I compiled a
new kernel without it and the LORs appears to be gone.

This particular machine has no serial port (laptop) so getting the full
output of show witness is a bit hard (is there a way to extract it from
a dump?).
I hand-typed some of the output, not sure how usefull it is. I have
another machine with a fxp interface (and a serial port) but it
needs to be running at the moment, I can probably "play" with it
this weekend.

0 fifo mutex -- last acquired _at_ /usr/src/sys/fs/fifofs/fifo_vnops.c:212
9  so_snd -- last acquired _at_ /usr/src/sys/kern/uipc_socket.c:1993
10  so_rcv -- last acquired _at_ /usr/src/sys/kern/uipc_socket.c:1994
11   sellck -- last acquired _at_ /usr/src/sys/kern/sys_generic.c:764
11   radix node head -- last acquired _at_ /usr/src/sys/net/route.c:148
12    rtentry -- last acquired _at_ /usr/src/sys/netinet/ip_route.c:830
13     ifaddr -- last acquired _at_ /usr/src/sys/net/route.c:791
13     rts_inq -- last acquired _at_ /usr/src/sys/net/netisr.c:232
13     if send queue -- last acquired _at_ /usr/src/sys/dev/fxp/if_fxp.c:1212
12    ifnet -- last acquired _at_ /usr/src/sys/net/if.c:1159

Also, while trying various things with dhclient (with
a DEVICE_POLLING-aware kernel), I got two other networking related LORs.

lock order reversal
  1st 0xc0810180 Giant (Giant) _at_ /usr/src/sys/kern/kern_descrip.c:1874
  2nd 0xc085f88c udp (udp) _at_ /usr/src/sys/netinet/udp_usrreq.c:1009
KDB: stack backtrace:
kdb_backtrace(c07a2861,c085f88c,c07a23ce,c07a23ce,c07acb9d) at 
kdb_backtrace+0x2e
witness_checkorder(c085f88c,9,c07acb9d,3f1,0) at witness_checkorder+0x5a4
_mtx_lock_flags(c085f88c,0,c07acb9d,3f1,c1d53a20) at _mtx_lock_flags+0x32
udp_detach(c2079858,c05a9550,246,c07df404,c1973304) at udp_detach+0x2b
soclose(c2079858,c079d789,847,c1d53a20,c1d53a20) at soclose+0x262
soo_close(c1d53a20,c1c3fc80,c079d789,847,c1d53a20) at soo_close+0x6d
fdrop_locked(c1d53a20,c1c3fc80,c079d789,832) at fdrop_locked+0x94
fdrop(c1d53a20,c1c3fc80,c079d789,77d,c05a9550,c079d789,c07a26ec,3,c1c3fc80,e5943bb0
,1,c079d789,e5943bac,c05a9cd6,c085bda0,c20f192c,246,c07df404,c20f192c,64a,c079d789,
e5943bd4,c0578b92,c20f192c,8,c079d789,64a) at fdrop+0x3c
closef(c1d53a20,c1c3fc80,c079d789,64a,c085bda0) at closef+0x3f2
fdfree(c1c3fc80,0,c079db0d,e6,0) at fdfree+0x526
exit1(c1c3fc80,100,e5943d30,c0754370,c1c3fc80) at exit1+0x4a7
sys_exit(c1c3fc80,e5943d04,4,6,1) at sys_exit+0x1d
syscall(3b,3b,3b,1,805670d) at syscall+0x2c0
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (1, FreeBSD ELF32, sys_exit), eip = 0x2814af73, esp = 
0xbfbfe3cc, ebp = 0xbfbfe3d8 ---

lock order reversal
  1st 0xc085ca40 bpf global lock (bpf global lock) _at_ 
/usr/src/sys/net/bpf.c:425
  2nd 0xc1b74018 fxp0 (network driver) _at_ /usr/src/sys/dev/fxp/if_fxp.c:2367
KDB: stack backtrace:
kdb_backtrace(c07a2861,c1b74018,c1b71520,c0785ad5,c0793032) at 
kdb_backtrace+0x2e
witness_checkorder(c1b74018,9,c0793032,93f,246) at witness_checkorder+0x5a4
_mtx_lock_flags(c1b74018,0,c0793032,93f,0) at _mtx_lock_flags+0x32
fxp_ioctl(c1b6f000,80206910,e593da38,c07a26ec,1) at fxp_ioctl+0x90
if_setflag(c1b6f000,100,20000,c1b6f044,0) at if_setflag+0x138
ifpromisc(c1b6f000,0,c07a6c65,14f,c2104100) at ifpromisc+0x3b
bpf_detachd(c2104100,0,c07a6c65,1a9,c1fea400) at bpf_detachd+0xeb
bpfclose(c1fea400,3,2000,c1c3f960,c2100440) at bpfclose+0xb4
giant_close(c1fea400,3,2000,c1c3f960,c1fea400) at giant_close+0x4f
devfs_close(e593db44,e593db70,c05f0626,c07d7f00,e593db44) at 
devfs_close+0x36b
VOP_CLOSE_APV(c07d7f00,e593db44,c1c3f960,c1c6a000,c0802940) at 
VOP_CLOSE_APV+0x3e
vn_close(c2100440,3,c1ff7a00,c1c3f960,68f) at vn_close+0x76
vn_closefile(c20d41f8,c1c3f960,e593dc04,c0561054,c20d41f8) at 
vn_closefile+0xf4
devfs_close_f(c20d41f8,c1c3f960,c079d789,847,c20d41f8) at devfs_close_f+0x19
fdrop_locked(c20d41f8,c1c3f960,c079d789,832) at fdrop_locked+0x94
fdrop(c20d41f8,c1c3f960,c079d789,77d,c0815d60,0,c07a2570,68f,c085bda4,e593dc7c,1,c0
85bda0,e593dc78,c05a9d07,0,c1e1162c,246,c07df404,c1e1162c,3ea,c079d789,e593dca0,c05
78b92,c1e1162c,8,c079d789,3ea) at fdrop+0x3c
closef(c20d41f8,c1c3f960,c079d789,3ea,c1c3f960) at closef+0x3f2
close(c1c3f960,e593dd04,4,c07b2fd7,1) at close+0x1f2
syscall(3b,3b,3b,0,8199000) at syscall+0x2c0
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (6, FreeBSD ELF32, close), eip = 0x282df9a7, esp = 
0xbfbfeabc, ebp = 0x
bfbfead8 ---


	Fredrik Lindberg