Silent hang in buildworld, was Re: Invoking -v for clang during buildworld

On Sat, Jan 16, 2021 at 03:04:04PM -0800, Mark Millard wrote:
> 
> Other than -j1 style builds (or equivalent), one pretty much
> always needs to go looking around for a non-panic failure. It
> is uncommon for all the material to be together in the build
> log in such contexts.

Running make cleandir twice and restarting -j4 buildworld brought
the process full circle: A silent hang, no debugger response, no
console warnings. That's what sent me down the rabbit hole of make
without clean, which worked at least once...

The residue of the top screen shows

last pid: 63377;  load averages:  4.29,  4.18,  4.15                         up 1+07:11:07  04:46:46
60 processes:  5 running, 55 sleeping
CPU: 70.7% user,  0.0% nice, 26.5% system,  2.8% interrupt,  0.0% idle
Mem: 631M Active, 4932K Inact, 92M Laundry, 166M Wired, 98M Buf, 18M Free
Swap: 2048M Total, 119M Used, 1928M Free, 5% Inuse, 16K In, 3180K Out
packet_write_wait: Connection to 50.1.20.26 port 22: Broken pipe
bob_at_raspberrypi:~ $ ssh www.zefox.com    RES STATE    C   TIME    WCPU COMMAND
ssh: connect to host www.zefox.com port 22: Connection timed out86.17% c++
bob_at_raspberrypi:~ $ 1  99    0   277M   231M RUN      0   3:26  75.00% c++
63245 bob           1  99    0   219M   173M CPU0     0   2:10  73.12% c++
62690 bob           1  98    0   354M   234M RUN      3   9:42  47.06% c++
63377 bob           1  30    0  5856K  2808K nanslp   0   0:00   3.13% gstat
38283 bob           1  24    0  5208K   608K wait     2   2:00   0.61% sh
  995 bob           1  20    0  6668K  1184K CPU3     3   8:46   0.47% top
  990 bob           1  20    0    12M  1060K select   2   0:48   0.05% sshd
....

[apologies for typing over the remnants]

I've put copies of the build and swap logs at

http://www.zefox.net/~fbsd/rpi2/buildworld/

The last vmstat entry (10 second repeat time) reports:
procs     memory       page                      disks     faults       cpu
r b w     avm     fre  flt  re  pi  po    fr   sr da0 sd0   in   sy   cs us sy id
 4  0 14  969160   91960   685   2   2   1   707  304   0   0 11418   692  1273 45  5 50

Does that point to the memory exhaustion suggested earlier in the thread?
At this point /boot/loader.conf contains vm.pfault_oom_attempts="-1", but 
that's a relic of long-ago attempts to use USB flash for root and swap.
Might removing it stimulate more warning messages?

Thanks for reading!

bob prohaska