Re: [rfc] migrate lagg to an rmlock

Sorry, I meant "line contention" rather than "lock contention". Yes, you're
right.



-adrian



On 24 August 2013 07:16, Robert Watson <rwatson_at_freebsd.org> wrote:

> On Sat, 24 Aug 2013, Alexander V. Chernikov wrote:
>
>  On 24.08.2013 00:54, Adrian Chadd wrote:
>>
>>>
>>> I'd like to commit this to -10. It migrates the if_lagg locking
>>> from a rw lock to a rm lock. We see a bit of contention between the
>>> transmit and
>>>
>>
>> We're running lagg with rmlock on several hundred heavily loaded
>> machines, it really works better. However, there should not be any
>> contention between receive and transmit side since there is actually no
>> _real_ need to lock RX (and even use lagg receive code at all):
>>
>> http://lists.freebsd.org/**pipermail/svn-src-all/2013-**April/067570.html<http://lists.freebsd.org/pipermail/svn-src-all/2013-April/067570.html>
>>
>
> We should distinguish "lock contention" from "line contention".  When
> acquiring a rwlock on multiple CPUs concurrently, the cache lines used to
> implement the lock are contended, as they must bounce between caches via
> the cache coherence protocol, also referred to as "contention".  In the
> if_lagg code, I assume that the read-only acquire of the rwlock (and
> perhaps now rmlock) is for data stability rather than mutual exclusion --
> e.g., to allow processing to completion against a stable version of the
> lagg configuration. As such, indeed, there should be no lock contention
> unless a configuration update takes place, and any line contention is a
> property of the locking primitive rather than data model.
>
> There are a number of other places in the kernel where migration to an
> rmlock makes sense -- however, some care must be taken for four reasons:
> (1) while read locks don't experience line contention, write locking
> becomes observably e.g., rmlocks might not be suitable for tcbinfo; (2)
> rmlocks, unlike rwlocks, more expensive so is not suitable for all rwlock
> line contention spots -- implement reader priority propagation, so you must
> reason about; and (3) historically, rmlocks have not fully implemented
> WITNESS so you may get less good debugging output.  if_lagg is a nice place
> to use rmlocks, as reconfigurations are very rare, and it's really all
> about long-term data stability.
>
> Robert
>