On 3/11/2018 1:43 PM, Jeff Roberson wrote: > Hi Folks, > > This could be my fault from recent NUMA and concurrency related work. I > did touch some of the arc back-pressure mechanisms. First, I would like > to identify whether the wired memory is in the buffer cache. Can those > of you that have a repro look at sysctl vfs.bufspace and tell me if that > accounts for the bulk of your wired memory usage? I'm wondering if a > job ran that pulled in all of the bufs from your root disk and filled up > the buffer cache which doesn't have a back-pressure mechanism. Then arc > didn't respond appropriately to lower its usage. > > Also, if you could try going back to r328953 or r326346 and let me know > if the problem exists in either. That would be very helpful. If anyone > is willing to debug this with me contact me directly and I will send > some test patches or debugging info after you have done the above steps. > > Thank you for the reports. > > Jeff It's like there is no ZFS ARC backpressure at all anymore... Mem: 8783M Active, 2851M Inact, 3716M Laundry, 53G Wired, 2449M Free ARC: 40G Total, 27G MFU, 11G MRU, 13M Anon, 631M Header, 955M Other 37G Compressed, 53G Uncompressed, 1.43:1 Ratio Swap: 116G Total, 18G Used, 98G Free, 15% Inuse, 4K In vfs.bufspace: 0 I ran out of swap space around 16GB and had to add a larger device on to keep a package test build going. Was there ever backpressure? Having to limit the arc size has been a constant workaround for package building. I have some large tmpfs at the moment totaling around ~28GB. > # df -ah|egrep '(Used|tmpfs)' > Filesystem Size Used Avail Capacity Mounted on > tmpfs 99G 520K 99G 0% /tmp > tmpfs 10G 8.0K 10G 0% /poudriere/data/.m/exp-11amd64-commit-test > tmpfs 10G 373M 9.6G 4% /poudriere/data/.m/exp-11amd64-commit-test/ref > tmpfs 1.0G 18M 1.0G 2% /poudriere/data/.m/exp-11amd64-commit-test/ref/.p > tmpfs 10G 176K 10G 0% /poudriere/data/.m/exp-11amd64-commit-test/ref/var/db/ports > tmpfs 10G 405M 9.6G 4% /poudriere/data/.m/exp-11amd64-commit-test/01 > tmpfs 10G 1.1G 8.9G 11% /poudriere/data/.m/exp-11amd64-commit-test/05 > tmpfs 10G 2.4G 7.6G 24% /poudriere/data/.m/exp-11amd64-commit-test/04 > tmpfs 10G 578M 9.4G 6% /poudriere/data/.m/exp-11amd64-commit-test/03 > tmpfs 10G 1.0G 9.0G 10% /poudriere/data/.m/exp-11amd64-commit-test/02 > tmpfs 10G 936M 9.1G 9% /poudriere/data/.m/exp-11amd64-commit-test/06 > tmpfs 10G 2.7G 7.3G 27% /poudriere/data/.m/exp-11amd64-commit-test/07 > tmpfs 10G 2.7G 7.3G 27% /poudriere/data/.m/exp-11amd64-commit-test/09 > tmpfs 10G 875M 9.1G 9% /poudriere/data/.m/exp-11amd64-commit-test/08 > tmpfs 10G 2.7G 7.3G 27% /poudriere/data/.m/exp-11amd64-commit-test/10 > tmpfs 10G 2.7G 7.3G 27% /poudriere/data/.m/exp-11amd64-commit-test/12 > tmpfs 10G 390M 9.6G 4% /poudriere/data/.m/exp-11amd64-commit-test/11 > tmpfs 10G 992M 9.0G 10% /poudriere/data/.m/exp-11amd64-commit-test/14 > tmpfs 10G 2.9G 7.1G 29% /poudriere/data/.m/exp-11amd64-commit-test/13 > tmpfs 1.0G 16K 1.0G 0% /poudriere/data/.m/exp-11amd64-commit-test/01/.p > tmpfs 1.0G 540K 1.0G 0% /poudriere/data/.m/exp-11amd64-commit-test/12/.p > tmpfs 1.0G 1.3M 1.0G 0% /poudriere/data/.m/exp-11amd64-commit-test/05/.p > tmpfs 1.0G 1.3M 1.0G 0% /poudriere/data/.m/exp-11amd64-commit-test/06/.p > tmpfs 1.0G 396K 1.0G 0% /poudriere/data/.m/exp-11amd64-commit-test/03/.p > tmpfs 1.0G 540K 1.0G 0% /poudriere/data/.m/exp-11amd64-commit-test/09/.p > tmpfs 1.0G 540K 1.0G 0% /poudriere/data/.m/exp-11amd64-commit-test/10/.p > tmpfs 1.0G 732K 1.0G 0% /poudriere/data/.m/exp-11amd64-commit-test/07/.p > tmpfs 1.0G 28K 1.0G 0% /poudriere/data/.m/exp-11amd64-commit-test/08/.p > tmpfs 1.0G 12K 1.0G 0% /poudriere/data/.m/exp-11amd64-commit-test/11/.p > tmpfs 1.0G 1.3M 1.0G 0% /poudriere/data/.m/exp-11amd64-commit-test/02/.p > tmpfs 1.0G 852K 1.0G 0% /poudriere/data/.m/exp-11amd64-commit-test/14/.p > tmpfs 1.0G 540K 1.0G 0% /poudriere/data/.m/exp-11amd64-commit-test/13/.p > tmpfs 1.0G 688K 1.0G 0% /poudriere/data/.m/exp-11amd64-commit-test/04/.p > tmpfs 10G 125M 9.9G 1% /poudriere/data/.m/exp-11amd64-commit-test/03/usr/local > tmpfs 10G 270M 9.7G 3% /poudriere/data/.m/exp-11amd64-commit-test/04/usr/local > tmpfs 10G 414M 9.6G 4% /poudriere/data/.m/exp-11amd64-commit-test/07/usr/local > tmpfs 10G 332M 9.7G 3% /poudriere/data/.m/exp-11amd64-commit-test/09/usr/local > tmpfs 10G 332M 9.7G 3% /poudriere/data/.m/exp-11amd64-commit-test/10/usr/local > tmpfs 10G 332M 9.7G 3% /poudriere/data/.m/exp-11amd64-commit-test/12/usr/local > tmpfs 10G 332M 9.7G 3% /poudriere/data/.m/exp-11amd64-commit-test/13/usr/local > tmpfs 10G 575M 9.4G 6% /poudriere/data/.m/exp-11amd64-commit-test/05/usr/local > tmpfs 10G 575M 9.4G 6% /poudriere/data/.m/exp-11amd64-commit-test/02/usr/local > tmpfs 10G 306M 9.7G 3% /poudriere/data/.m/exp-11amd64-commit-test/14/usr/local > tmpfs 10G 570M 9.4G 6% /poudriere/data/.m/exp-11amd64-commit-test/06/usr/local > tmpfs 10G 36M 10G 0% /poudriere/data/.m/exp-11amd64-commit-test/01/usr/local > tmpfs 10G 283M 9.7G 3% /poudriere/data/.m/exp-11amd64-commit-test/11/usr/local > tmpfs 10G 301M 9.7G 3% /poudriere/data/.m/exp-11amd64-commit-test/08/usr/local And some large processes. > # ps aux|sort -rnk 6|head -n 20 > root 2481 100.0 4.3 3334388 3209728 15 RJ 14:40 119:54.18 /usr/local/bin/python2.7 ../../rpython/bin/rpython --source -Ojit targetpypystandalone.py > root 46466 97.7 0.8 620692 581304 15 RJ 16:44 0:36.07 /wrkdirs/usr/ports/lang/ghc/work/ghc-8.0.2/inplace/lib/bin/ghc-stage1 -B/wrkdirs/usr/ports/lang/ghc/work/ghc-8.0.2/inplace/lib -hisuf hi -osuf o -hcsuf hc -static -H32m -O -I/usr/include -L/usr/lib -I/usr/loc > root 44667 100.0 0.2 187664 161444 15 RJ 16:43 1:04.16 /wrkdirs/usr/ports/lang/gcc6/work/.build/./prev-gcc/cc1plus -quiet -nostdinc++ -I . -I c-family -I /wrkdirs/usr/ports/lang/gcc6/work/gcc-6.4.0/gcc -I /wrkdirs/usr/ports/lang/gcc6/work/gcc-6.4.0/gcc/c-family - > root 47917 76.5 0.2 173572 140580 15 RJ 16:44 0:12.45 /usr/bin/c++ -cc1 -triple x86_64-unknown-freebsd11.1 -emit-obj -disable-free -disable-llvm-verifier -discard-value-names -main-file-name MemorySanitizer.cpp -mrelocation-model pic -pic-level 2 -mthread-model > root 47918 75.9 0.2 164784 136976 15 RJ 16:44 0:12.50 /wrkdirs/usr/ports/lang/gcc7-devel/work/.build/./prev-gcc/cc1plus -quiet -nostdinc++ -I . -I c -I /wrkdirs/usr/ports/lang/gcc7-devel/work/gcc-7-20180308/gcc -I /wrkdirs/usr/ports/lang/gcc7-devel/work/gcc-7-20 > root 47938 71.5 0.2 159952 134260 15 RJ 16:44 0:11.17 /wrkdirs/usr/ports/lang/gcc6-devel/work/.build/./prev-gcc/cc1plus -quiet -nostdinc++ -I . -I c-family -I /wrkdirs/usr/ports/lang/gcc6-devel/work/gcc-6-20180307/gcc -I /wrkdirs/usr/ports/lang/gcc6-devel/work/g > root 48090 48.1 0.2 159420 125748 15 RJ 16:44 0:06.02 /usr/bin/c++ -cc1 -triple x86_64-unknown-freebsd11.1 -emit-obj -disable-free -disable-llvm-verifier -discard-value-names -main-file-name AlignmentFromAssumptions.cpp -mrelocation-model pic -pic-level 2 -mthre > root 48110 38.4 0.2 153988 116220 15 RJ 16:44 0:04.35 /usr/bin/c++ -cc1 -triple x86_64-unknown-freebsd11.1 -emit-obj -disable-free -disable-llvm-verifier -discard-value-names -main-file-name property-type.cxx -mrelocation-model static -mthread-model posix -mdisa > root 48139 35.4 0.1 131636 101608 15 RJ 16:44 0:03.70 /usr/bin/c++ -cc1 -triple x86_64-unknown-freebsd11.1 -emit-obj -disable-free -disable-llvm-verifier -discard-value-names -main-file-name InstrInfoEmitter.cpp -mrelocation-model pic -pic-level 2 -mthread-model > root 48119 34.2 0.1 126396 93768 15 RJ 16:44 0:03.79 /usr/bin/c++ -cc1 -triple x86_64-unknown-freebsd11.1 -emit-obj -disable-free -disable-llvm-verifier -discard-value-names -main-file-name NaryReassociate.cpp -mrelocation-model pic -pic-level 2 -mthread-model -- Regards, Bryan DreweryReceived on Fri Mar 16 2018 - 22:46:12 UTC
This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:41:15 UTC