Re: UMA cache back pressure

On 18.11.2013 10:41, Adrian Chadd wrote:
> Your patch does three things:
>
> * adds a couple new buckets;

These new buckets make bucket size self-tuning more soft and precise. 
Without them there are buckets for 1, 5, 13, 29, ... items. While at 
bigger sizes difference about 2x is fine, at smallest ones it is 5x and 
2.6x respectively. New buckets make that line look like 1, 3, 5, 9, 13, 
29, reducing jumps between steps, making algorithm work softer, 
allocating and freeing memory in better fitting chunks. Otherwise there 
is quite a big gap between allocating 128K and 5x128K of RAM at once.

> * reduces some lock contention

More precisely patch adds check for congestion on free to grow bucket 
sizes same as on allocation. As consequence that indeed should reduce 
lock congestion, but I don't have specific numbers. All I see is that VM 
and UMA mutexes no longer appear in profiling top after all these changes.

* does soft back pressure

In this list you have missed mentioning small but major point of the 
patch -- we should prevent problems, not just solve them. As I have 
written in original email, this specific change shown me 1.5x 
performance improvement in low-memory condition. As I understand, that 
happened because VM no longer have to repeatedly allocate and free 
hugely oversized buckets of 10-15 * 128K.

> * does the aggressive backpressure.

After all above that is mostly just a safety belt. With 40GB RAM that 
code was triggered only couple times during full hour of testing with 
debug logging inserted there. On machine with 2GB RAM it is triggered 
quite regularly and probably that is unavoidable since even with lowest 
bucket size of one item 24 CPUs mean 48 cache buckets, i.e. up to 6MB of 
otherwise unreleasable memory for single 128K zone.

> So, do you get any benefits from just the first one, or first two?

I don't see much reason to handle that in pieces. As I have described 
above, each part has own goal, but they much better work together.

> On 17 November 2013 15:09, Alexander Motin <mav_at_freebsd.org> wrote:
>> Hi.
>>
>> I've created patch, based on earlier work of avg_at_, to add back pressure to
>> UMA allocation caches. The problem of physical memory or KVA exhaustion
>> existed there for many years and it is quite critical now for improving
>> systems performance while keeping stability. Changes done in memory
>> allocation last years improved situation. but haven't fixed completely. My
>> patch solves remaining problems from two sides: a) reducing bucket sizes
>> every time system detects low memory condition; and b) as last-resort
>> mechanism for very low memory condition, it cycling over all CPUs to purge
>> their per-CPU UMA caches. Benefit of this approach is in absence of any
>> additional hard-coded limits on cache sizes -- they are self-tuned, based on
>> load and memory pressure.
>>
>> With this change I believe it should be safe enough to enable UMA allocation
>> caches in ZFS via vfs.zfs.zio.use_uma tunable (at least for amd64). I did
>> many tests on machine with 24 logical cores (and as result strong allocation
>> cache effects), and can say that with 40GB RAM using UMA caches, allowed by
>> this change, by two times increases results of SPEC NFS benchmark on ZFS
>> pool of several SSDs. To test system stability I've run the same test with
>> physical memory limited to just 2GB and system successfully survived that,
>> and even showed results 1.5 times better then with just last resort measures
>> of b). In both cases tools/umastat no longer shows unbound UMA cache growth,
>> that makes me believe in viability of this approach for longer runs.
>>
>> I would like to hear some comments about that:
>> http://people.freebsd.org/~mav/uma_pressure.patch


-- 
Alexander Motin