Re: [rfc] migrate lagg to an rmlock

On Sat, Aug 24, 2013 at 7:16 AM, Robert Watson <rwatson_at_freebsd.org> wrote:
> On Sat, 24 Aug 2013, Alexander V. Chernikov wrote:
>
>> On 24.08.2013 00:54, Adrian Chadd wrote:
>>>
>>>
>>> I'd like to commit this to -10. It migrates the if_lagg locking
>>> from a rw lock to a rm lock. We see a bit of contention between the
>>> transmit and
>>
>>
>> We're running lagg with rmlock on several hundred heavily loaded machines,
>> it really works better. However, there should not be any contention between
>> receive and transmit side since there is actually no _real_ need to lock RX
>> (and even use lagg receive code at all):
>>
>> http://lists.freebsd.org/pipermail/svn-src-all/2013-April/067570.html
>
>
> We should distinguish "lock contention" from "line contention".  When
> acquiring a rwlock on multiple CPUs concurrently, the cache lines used to
> implement the lock are contended, as they must bounce between caches via the
> cache coherence protocol, also referred to as "contention".  In the if_lagg
> code, I assume that the read-only acquire of the rwlock (and perhaps now
> rmlock) is for data stability rather than mutual exclusion -- e.g., to allow
> processing to completion against a stable version of the lagg configuration.
> As such, indeed, there should be no lock contention unless a configuration
> update takes place, and any line contention is a property of the locking
> primitive rather than data model.
>
> There are a number of other places in the kernel where migration to an
> rmlock makes sense -- however, some care must be taken for four reasons: (1)
> while read locks don't experience line contention, write locking becomes
> observably e.g., rmlocks might not be suitable for tcbinfo; (2) rmlocks,
> unlike rwlocks, more expensive so is not suitable for all rwlock line
> contention spots -- implement reader priority propagation, so you must
> reason about; and (3) historically, rmlocks have not fully implemented
> WITNESS so you may get less good debugging output.  if_lagg is a nice place

I'm not sure what you mean here with (3), because from my
understanding of the code WITNESS is implemented both in the sleepable
and non-sleepable case, but there could be something I'm missing.
Something I think we lack in rmlock code is fully supported
LOCK_PROFILING as we have in all the other primitives, but again, if
I'm wrong feel free to correct me.

> to use rmlocks, as reconfigurations are very rare, and it's really all about
> long-term data stability.
>
> Robert
>
> _______________________________________________
> freebsd-current_at_freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-current
> To unsubscribe, send any mail to "freebsd-current-unsubscribe_at_freebsd.org"

Thanks,

-- 
Davide

"There are no solved problems; there are only problems that are more
or less solved" -- Henri Poincare