Re: FYI: what it takes for RAM+swap to build devel/llvm40 with 4 processors or cores and WITH__DEBUG= (powerpc64 example)

[I add some what-it-take-for-an-upgrade information.]

On 2017-Mar-12, at 6:53 PM, Mark Millard <markmi_at_dsl-only.net> wrote:

> Summary: RAM+(peak swap) was about 26 GiBytes.
>         Also: about 118 GiByte /usr/obj/. . ./llvm40/ area.
>         (2 processors, 2 cores each, all in use;
>          WITH_DEBUG= used)
> 
> The peak usage times were when the 4 cores were
> each busy running ld at the same time.
> 
> [So far as I know FreeBSD does not report peak swap usage
> "since boot". So I do not have a cross check on if I missed
> seeing a higher peak then I report in the details below.]
> 
> What all this note spans as part of the build:
> 
> # more /var/db/ports/devel_llvm40/options
> # This file is auto-generated by 'make config'.
> # Options for llvm40-4.0.0.r4
> _OPTIONS_READ=llvm40-4.0.0.r4
> _FILE_COMPLETE_OPTIONS_LIST=CLANG DOCS EXTRAS LIT LLD LLDB
> OPTIONS_FILE_SET+=CLANG
> OPTIONS_FILE_SET+=DOCS
> OPTIONS_FILE_SET+=EXTRAS
> OPTIONS_FILE_SET+=LIT
> OPTIONS_FILE_SET+=LLD
> OPTIONS_FILE_SET+=LLDB
> 
> The system clang 4.0 was used to do the build. A port
> binutils was used (-B${LOCALBASE}/bin/ in CFLAGS,
> CXXFLAGS, an CPPFLAGS). The kernel was non-debug generally
> but buildworld buildkernel did not have MALLOC_PRODUCTION= .
> The llvm40 build did have MALLOC_PRODUCTION= .
> 
> # uname -paKU
> FreeBSD FBSDG5L 12.0-CURRENT FreeBSD 12.0-CURRENT  r314687M  powerpc powerpc64 1200023 1200023
> 
> Most of what I have access to for FreeBSD does not have a
> big enough configuration to do a WITH_DEBUG= build of llvm40
> on a machine with 4 cores, all in use.
> 
> One type of environment that does is an old PowerMac G5
> so-called "Quad Core" that has 16 GiBytes of RAM, 17 GiBytes
> of swap, and a 480 GiByte SSD (but extra over provisioned so
> it appears even smaller for the file system+swap).
> 
> Watching with top the peak swap usage that I saw was
> 56% of the 17 GiByte --so call it 10 GiBytes or so.
> 
> So something like 16 GiBytes RAM + 10 GiBytes swap
> and so something like 26 GiByte total.
> 
> I used portmaster with -DK. Afterwards the /usr/obj/
> sub-area for llvm40 used totaled to a size of:
> 
> # du -sg /usr/obj/portswork/usr/ports/devel/llvm40
> 118	/usr/obj/portswork/usr/ports/devel/llvm40
> 
> So around 118 GiBytes of disk space.
> 
> Showing the major space usage contributions:
> 
> # du -sg /usr/obj/portswork/usr/ports/devel/llvm40/work/.build/* /usr/obj/portswork/usr/ports/devel/llvm40/work/stage/usr/local/llvm40/*
> . . .
> 29	/usr/obj/portswork/usr/ports/devel/llvm40/work/.build/bin
> . . .
> 29	/usr/obj/portswork/usr/ports/devel/llvm40/work/.build/lib
> . . .
> 12	/usr/obj/portswork/usr/ports/devel/llvm40/work/.build/tools
> . . .
> 26	/usr/obj/portswork/usr/ports/devel/llvm40/work/stage/usr/local/llvm40/bin
> . . .
> 24	/usr/obj/portswork/usr/ports/devel/llvm40/work/stage/usr/local/llvm40/lib
> . . .
> 
> 
> 
> Side notes that are more system specific:
> 
> The timestamps on the script output file indicate that
> the build took about 8 hours 24 minutes.
> 
> The powerpc64 system used was built with the system clang
> 4.0 compiler and a port-based binutils. This is despite that
> clang 4.0 produces code that has any thrown C++ exceptions
> completely non-functional for powerpc64 (program crashes
> via signals reporting problems).



I upgraded from llvm40 r4 to final. An interesting result was
its creation of a backup package for llvm40-4.0.0.r4:

about 13 cpu-core-hours running pkg create

(Remember: I've been building with WITH_DEBUG= ) Its
single-threaded status stands out via elapsed time
approximately matching.

I'll note that it was somewhat under 6 elapsed hours for
staging to have been populated (-j4 with 4 cores present
helps for this part).

(Of course these elapsed-time figures are rather system
dependent, although the ratio might be more stable.)



Also interesting was:

Installed packages to be REMOVED:
	llvm40-4.0.0.r4

Number of packages to be removed: 1

The operation will free 49 GiB.


===
Mark Millard
markmi at dsl-only.net