Re: Silent hang in buildworld, was Re: Invoking -v for clang during buildworld

From: Mark Millard <marklmi_at_yahoo.com>
Date: Mon, 18 Jan 2021 21:12:43 -0800
On 2021-Jan-18, at 19:19, Mark Millard <marklmi at yahoo.com> wrote:

> . . .
>> FYI: I re-established my access to a RPi2B V1.1 and made
>> it report: "maximum recommended amount (468832 pages)"
>> 
>> (The figure can vary some from release to release.)
>> 
>> 468832*4096 == 1920335872 or a little over 1831 MiBytes
>> 
>> For the 4096 Byte pages, that means that the following from
>> gpart fits without complaint (size is in blocks, not pages):
>> 
>> 413140992    3686400  da0p2  freebsd-swap  (1.8G)
>> 
>> 3686400*512 is a little over 1.75 GiByte or 1800 MiByte. So
>> I've left some room below 1831 MiBytes, but not a lot.
>> 
>> FYI about my build experiment that is running:
>> 
>> # sysctl hw.physmem
>> hw.physmem: 979042304
>> 
>> which, in recent times for armv7, I can (and did) set in
>> /boot/loader.conf on a faster cortex-A7 SBC (that can boot
>> the same media but has more RAM).
>> 
>> So I tried a -j4 build, but with LDFLAGS.lld+= -Wl,--threads=1
>> in use and my other particular src.conf/make.conf like content
>> (so the builds do likely differ from yours in various ways).
>> My build is producing a non-debug build (but with -g symbols).
>> Somewhat after where your buildworld.log stops, my odd variant
>> of top was reporting:
>> 
>> Mem:  . . . , 753672Ki MaxObsActive, 200412Ki MaxObsWired, 892732Ki MaxObs(Act+Wir)
>> Swap: . . . , 145832Ki MaxObsUsed
>> 
>> and top was also showing lots of processes as having "0B" RES
>> in STATE "wait" or "nanslp" (so, apparently swapped out, not paging).
>> ("MaxObs" is short for "maximum observed".)
>> 
>> For comparison, your swapscript.log reported a maximum total of
>> 346192 KiBytes "Used" for swap, about 98% into the log file.
>> 
>> (Time goes by . . .)
>> 
>> It finished with building libllvm and is part way into building
>> libclang. This is probably well past where your hangup happened,
>> given that your published buildworldlog file stopped with
>> libllvm's Target/ARM/ARMMCInstLower.o . My odd top now shows:
>> 
>> Mem:  . . . , 753672Ki MaxObsActive, 200412Ki MaxObsWired, 892732Ki MaxObs(Act+Wir)
>> Swap: . . . , 392328Ki MaxObsUsed
>> 
>> The build continues to run. I'll let you know how it goes.
>> . . .
> 
> Just after libclang finished my odd top showed:
> 
> Mem:  . . . , 753672Ki MaxObsActive, 200412Ki MaxObsWired, 892736Ki MaxObs(Act+Wir)
> Swap: . . . , 537588Ki MaxObsUsed
> 
> After liblldb:
> 
> Mem:  . . . , 753672Ki MaxObsActive, 200412Ki MaxObsWired, 899276Ki MaxObs(Act+Wir)
> Swap: . . . , 537588Ki MaxObsUsed
> 
> Much later, after the lldb program had been built:
> 
> Mem:  . . . , 765700Ki MaxObsActive, 200412Ki MaxObsWired, 954116Ki MaxObs(Act+Wir)
> Swap: . . . , 537588Ki MaxObsUsed
> 
>>>> World build completed on Mon Jan 18 19:10:08 PST 2021
>>>> World built in 72960 seconds, ncpu: 4, make -j4
> 
> This was building from scratch what was already installed:
> 
> # ~/fbsd-based-on-what-freebsd-main.sh 
> merge-base: 818390ce0ca539300dd15d7a817784f1e3f7a9b8
> merge-base: CommitDate: 2021-01-13 21:27:44 +0000
> 4180404713ec (HEAD -> mm-src) mm-src snapshot for mm's patched build in git context.
> 818390ce0ca5 (freebsd/main, freebsd/HEAD, pure-src, main) arm64: fix early devmap assertion
> FreeBSD OPiP2E_RPi2v11 13.0-CURRENT FreeBSD 13.0-CURRENT mm-src-c255938-g4180404713ec GENERIC-NODBG  arm armv7 1300135 1300135
> 
> This suggests that you should be able to build on the RPi2B v1.1,
> using -j4, with appropriate configuration for what and how to build.
> 
> 
> It is now building the matching kernel, my GENERIC-NODBG style.

Done:

>>> Kernel build for GENERIC-NODBG completed on Mon Jan 18 20:33:26 PST 2021
>>> Kernel(s)  GENERIC-NODBG built in 4998 seconds, ncpu: 4, make -j4

So, World+Kernel in somewhat under 22 hours.

The "MaxObs*" figures were unchanged, so:

Mem:  . . . , 765700Ki MaxObsActive, 200412Ki MaxObsWired, 954116Ki MaxObs(Act+Wir)
Swap: . . . , 537588Ki MaxObsUsed

This suggests that, for now, 800 MiByte of swap would be something
more than 1.5 times what it actually used and 1050 MiBytes would
be something like 2.0 times what it actually used, so leaving some
notable margin for variations in peek usage, at least when linker
threading is avoided.



As for what I used to control "what and how to build" . . .

# more ~/sys_build_scripts.armv7-host/make_armv7_nodebug_clang_bootstrap-armv7-host.sh 
kldload -n filemon && \
script ~/sys_typescripts/typescript_make_armv7_nodebug_clang_bootstrap-armv7-host-$(date +%Y-%m-%d:%H:%M:%S) \
env __MAKE_CONF="/root/src.configs/make.conf" SRCCONF="/dev/null" SRC_ENV_CONF="/root/src.configs/src.conf.armv7-clang-bootstrap.armv7-host" \
WITH_META_MODE=yes \
WORLD_FLAGS="${WORLD_FLAGS} UBLDR_LOADADDR=0x42000000" \
MAKEOBJDIRPREFIX="/usr/obj/armv7_clang/arm.armv7" \
make $*

(In my context, UBLDR_LOADADDR is ignored by anything that
can not use the figure given. So I've no bothered to be
more selective about having it in the armv7 builds.)

# more ~/src.configs/make.conf
LDFLAGS.lld+= -Wl,--threads=1

# more ~/src.configs/src.conf.armv7-clang-bootstrap.armv7-host
TO_TYPE=armv7
#
KERNCONF=GENERIC-NODBG
TARGET=arm
.if ${.MAKE.LEVEL} == 0
TARGET_ARCH=${TO_TYPE}
.export TARGET_ARCH
.endif
#
#WITH_CROSS_COMPILER=
WITH_SYSTEM_COMPILER=
WITH_SYSTEM_LINKER=
#
WITH_LIBCPLUSPLUS=
WITHOUT_BINUTILS_BOOTSTRAP=
WITH_ELFTOOLCHAIN_BOOTSTRAP=
#Disables avoiding bootstrap: WITHOUT_LLVM_TARGET_ALL=
WITHOUT_LLVM_TARGET_AARCH64=
WITH_LLVM_TARGET_ARM=
WITHOUT_LLVM_TARGET_MIPS=
WITHOUT_LLVM_TARGET_POWERPC=
WITHOUT_LLVM_TARGET_RISCV=
WITHOUT_LLVM_TARGET_X86=
WITH_CLANG=
WITH_CLANG_IS_CC=
WITH_CLANG_FULL=
WITH_CLANG_EXTRAS=
WITH_LLD=
WITH_LLD_IS_LD=
WITHOUT_BINUTILS=
#
WITH_LLDB=
#
WITH_BOOT=
WITHOUT_LIB32=
#
#
WITHOUT_WERROR=
#WERROR=
MALLOC_PRODUCTION=
WITH_MALLOC_PRODUCTION=
WITHOUT_ASSERT_DEBUG=
WITHOUT_LLVM_ASSERTIONS=
#
# Avoid stripping but do not control host -g status as well:
DEBUG_FLAGS+=
#
WITH_REPRODUCIBLE_BUILD=
WITH_DEBUG_FILES=
#
# Use of the .clang 's here avoids
# interfering with other C<?>FLAGS
# usage, such as ?= usage.
CFLAGS.clang+= -mcpu=cortex-a7
CXXFLAGS.clang+= -mcpu=cortex-a7
CPPFLAGS.clang+= -mcpu=cortex-a7

(I do not claim that you would want WITH_REPRODUCIBLE_BUILD .
I just happen to have been experimenting with it. You might
not want to be explicit about the cpu to target. You might
not want WITH_CLANG_EXTRAS .)

# more /usr/fbsd/mm-src/sys/arm/conf/GENERIC-NODBG
include "GENERIC"

ident   GENERIC-NODBG

makeoptions     DEBUG=-g                # Build kernel with gdb(1) debug symbols

options         AUDIT                   # Not enabled by default in armv7/v6 kernels
                                        # Enabled here to allow kyua test runs to
                                        # possibly report auditing works.

options         ALT_BREAK_TO_DEBUGGER

options         KDB                     # Enable kernel debugger support

# For minimum debugger support (stable branch) use:
options         KDB_TRACE               # Print a stack trace for a panic
options         DDB                     # Enable the kernel debugger

# Extra stuff:
#options        VERBOSE_SYSINIT=0       # Enable verbose sysinit messages
#options        BOOTVERBOSE=1
#options        BOOTHOWTO=RB_VERBOSE
options         ALT_BREAK_TO_DEBUGGER   # Enter debugger on keyboard escape sequence
options         KLD_DEBUG
#options        KTR
#options        KTR_MASK=KTR_TRAP
##options       KTR_CPUMASK=0xF
#options        KTR_VERBOSE

# Disable any extra checking for. . .
nooptions       INVARIANTS              # Enable calls of extra sanity checking
nooptions       INVARIANT_SUPPORT       # Extra sanity checks of internal structures, required by INVARIANTS
nooptions       WITNESS                 # Enable checks to detect deadlocks and cycles
nooptions       WITNESS_SKIPSPIN        # Don't run witness on spinlocks for speed
nooptions       DEADLKRES               # Enable the deadlock resolver
nooptions       MALLOC_DEBUG_MAXZONES   # Separate malloc(9) zones
nooptions       DIAGNOSTIC
nooptions       BUF_TRACKING
nooptions       FULL_BUF_TRACKING
nooptions       USB_DEBUG
nooptions       USB_REQ_DEBUG
nooptions       USB_VERBOSE

The /boot/loader.conf file and the /etc/sysctl.conf files
both contained:

vm.pageout_oom_seq=120
vm.pfault_oom_attempts=-1

(The hw.physmem=979042304 in /boot/loader.conf was very-special,
to better approximate your environment. I also controlled the
cpu frequency used via a line in /etc/sysctl.conf . I do not
bother with such non-default frequency usage [or related settings]
for RPi*'s with the pre-RPi4B style power connections but do
control the frequency for the OPi+2E.)


===
Mark Millard
marklmi at yahoo.com
( dsl-only.net went
away in early 2018-Mar)
Received on Tue Jan 19 2021 - 04:13:01 UTC

This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:41:26 UTC