Re: [rfc] migrate lagg to an rmlock

On 8/24/13 7:16 AM, Robert Watson wrote:
> On Sat, 24 Aug 2013, Alexander V. Chernikov wrote:
>
>> On 24.08.2013 00:54, Adrian Chadd wrote:
>>>
>>> I'd like to commit this to -10. It migrates the if_lagg locking
>>> from a rw lock to a rm lock. We see a bit of contention between the
>>> transmit and
>>
>> We're running lagg with rmlock on several hundred heavily loaded 
>> machines, it really works better. However, there should not be any 
>> contention between receive and transmit side since there is actually 
>> no _real_ need to lock RX (and even use lagg receive code at all):
>>
>> http://lists.freebsd.org/pipermail/svn-src-all/2013-April/067570.html
>
> We should distinguish "lock contention" from "line contention". When 
> acquiring a rwlock on multiple CPUs concurrently, the cache lines used 
> to implement the lock are contended, as they must bounce between 
> caches via the cache coherence protocol, also referred to as 
> "contention".  In the if_lagg code, I assume that the read-only 
> acquire of the rwlock (and perhaps now rmlock) is for data stability 
> rather than mutual exclusion -- e.g., to allow processing to 
> completion against a stable version of the lagg configuration. As 
> such, indeed, there should be no lock contention unless a 
> configuration update takes place, and any line contention is a 
> property of the locking primitive rather than data model.
>
> There are a number of other places in the kernel where migration to an 
> rmlock makes sense -- however, some care must be taken for four 
> reasons: (1) while read locks don't experience line contention, write 
> locking becomes observably e.g., rmlocks might not be suitable for 
> tcbinfo; (2) rmlocks, unlike rwlocks, more expensive so is not 
> suitable for all rwlock line contention spots -- implement reader 
> priority propagation, so you must reason about; and (3) historically, 
> rmlocks have not fully implemented WITNESS so you may get less good 
> debugging output.  if_lagg is a nice place to use rmlocks, as 
> reconfigurations are very rare, and it's really all about long-term 
> data stability.
>
> Robert
> _______________________________________________
> freebsd-net_at_freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "freebsd-net-unsubscribe_at_freebsd.org"
>

Robert, what do you think about a quick swap of the ifnet structures to 
counter before 10.x?

-Alfred

-- 
Alfred Perlstein