Re: [PATCH] Convert the VFS cache lock to an rmlock

On Fri, Mar 13, 2015 at 11:23:06AM -0400, Ryan Stone wrote:
> On Thu, Mar 12, 2015 at 1:36 PM, Mateusz Guzik <mjguzik_at_gmail.com> wrote:
> 
> > Workloads like buildworld and the like (i.e. a lot of forks + execs) run
> > into very severe contention in vm, which is orders of magnitude bigger
> > than anything else.
> >
> > As such your result seems quite suspicious.
> >
> 
> You're right, I did mess up the testing somewhere (I have no idea how).  As
> you suggested, I switched to using a separate partition for the objdir, and
> ran each build with a freshly newfsed filesystem.  I scripted it to be sure
> that I was following the same procedure with each run:
> 
[..]
> The difference is due to a significant increase in system time.  Write
> locks on an rmlock are extremely expensive (they involve an
> smp_rendezvous), and the cost likely scales with the number of cores:
> 
> x 32core/orig.log
> + 32core/rmlock.log
> +--------------------------------------------------------------------------+
> |xxx   x                                                 +   +++          +|
> ||_MA__|                                                 |____MA______|    |
> +--------------------------------------------------------------------------+
>     N           Min           Max        Median           Avg        Stddev
> x   5       5616.63        5715.7        5641.5       5661.72     48.511545
> +   5       6502.51       6781.84        6596.5       6612.39     103.06568
> Difference at 95.0% confidence
>         950.67 +/- 117.474
>         16.7912% +/- 2.07489%
>         (Student's t, pooled s = 80.5478)
> 
> 
> At this point I'm pretty much at an impasse.  The real-time behaviour is
> critical to me, but a 5% performance degradation isn't likely to be
> acceptable to many people.  I'll see what I can do with this.

Well, you can see that the entire cache is protected by one lock.

It also has other shortcomings which basically ask for a complete
rewrite, so I would say spending significant amount of time on it is not
worth discouraged.

Brief look suggests that while just protecting each hash bucket by a
separate lock may require a lot of work, it should be quite doable to do it
with an additional lock which makes deletions exclusive, while insertions
and lookups take the same lock shared and then lock specific buckets.

Also cache_purge could be micro-optimized to not take the lock if we did
not have any entries for given vnode (and we don't in a lot of cases).

With this in place I would not be surprised if the thing was faster even
with rmlock, which would justify it until the cache is rewritten.

-- 
Mateusz Guzik <mjguzik gmail.com>