Re: arc_reclaim_thread running hot

On Thu, 13 Sep 2018 at 15:19, Eric van Gyzen <eric_at_vangyzen.net> wrote:
>
> This morning, I found the arc_reclaim_thread running hot on my laptop
> running 12.0-ALPHA5 r338572.
>
> vfs.zfs.arc_max="4294967296"  <-- 4 GiB
>
> last pid: 13288;  load averages:  1.32,  1.26,  1.16
>
> Mem: 456M Active, 3837M Inact, 743M Laundry, 2563M Wired, 167M Free
> ARC: 1131M Total, 304M MFU, 145M MRU, 1344K Anon, 9116K Header, 671M Other
>       89M Compressed, 361M Uncompressed, 4.03:1 Ratio
>
>     22 root         -8    -      0   256K CPU2     2 309:20  99.75%
> zfskern{arc_reclaim_thread}
>
> zfs_arc_meta_strategy is still the default of 1.
>
> I sampled the thread's stacks with
>
> for N in `jot 1000`; do procstat -kk 100101; done | grep 100101
>
> and put the results here:
>
> https://people.freebsd.org/~vangyzen/arc_reclaim_thread_stacks.txt
>
> I'm happy to help debug this.  Just let me know what you need.
>
> Eric

Just to summarize, these are the top 15 active
(guessing that the leftmost function is "current"):
     22 arc_space_return
     22 uma_dbg_free
     26 uma_dbg_alloc
     30 mi_switch
     31 uma_zalloc_arg
     33 multilist_sublist_lock
     34 __mtx_unlock_flags
     39 <running>
     46 _sx_xlock
     48 hdr_full_cons
     50 uma_zfree_arg
     58 aggsum_add
     91 arc_adjust
    156 _sx_xunlock
    185 arc_evict_state

And the top 15 in total
(that is, no matter where in the call stack):
     44 multilist_sublist_lock
     44 uma_dbg_free
     46 _sx_xlock
     47 uma_dbg_alloc
     61 arc_space_return
    127 aggsum_add
    139 hdr_full_cons
    156 _sx_xunlock
    211 uma_zfree_arg
    219 uma_zalloc_arg
    833 arc_evict_state
    926 arc_adjust
    961 arc_reclaim_thread
    961 fork_exit
    961 fork_trampoline

(I have nothing sensible to add about the actual problem.)

-- 
Daniel Nebdal