Re: FYI: what it takes for RAM+swap to build devel/llvm40 with 4 processors or cores and WITH__DEBUG= (powerpc64 example)

From: Mark Millard <markmi_at_dsl-only.net>
Date: Mon, 3 Apr 2017 11:38:50 -0700
On 2017-Apr-1, at 3:51 AM, Mark Millard <markmi at dsl-only.net> wrote:

> On 2017-Mar-31, at 4:51 PM, Mark Millard <markmi at dsl-only.net> wrote:
> 
>> On 2017-Mar-30, at 7:51 PM, Mark Millard <markmi at dsl-only.net> wrote:
>> 
>>> On 2017-Mar-30, at 1:22 PM, Mark Millard <markmi at dsl-only.net> wrote:
>>> 
>>>> Sounds like the ALLOW_OPTIMIZATIONS_FOR_WITH_DEBUG technique
>>>> would not change the "WITNESS and INVARIANTS"-like part of the
>>>> issue. In fact if WITH_DEBUG= causes the cmake debug-style
>>>> llvm40 build ALLOW_OPTIMIZATIONS_FOR_WITH_DEBUG might not
>>>> make any difference: separate enforcing of lack of optimization.
>>>> 
>>>> But just to see what results I've done "pkg delete llvm40"
>>>> and am doing another build with ALLOW_OPTIMIZATIONS_FOR_WITH_DEBUG=
>>>> and its supporting code in place in addition to using WITH_DEBUG=
>>>> as the type of build fro FreeBSD's viewpoint.
>>>> 
>>>> If you know that the test is a waste of machine cycles, you can
>>>> let me know if you want.
>>> 
>>> The experiment showed that ALLOW_OPTIMIZATIONS_FOR_WITH_DEBUG
>>> use made no difference for devel/llvm40 so devel/llvm40 itself
>>> has to change such as what Dimitry Andric reported separately
>>> as a working change to the Makefile .
>>> 
>>> (ALLOW_OPTIMIZATIONS_FOR_WITH_DEBUG would still have its uses
>>> for various other ports.)
>> 
>> I've now tried with both ALLOW_OPTIMIZATIONS_FOR_WITH_DEBUG and:
> 
> I may have had a textual error that prevented
> ALLOW_OPTIMIZATIONS_FOR_WITH_DEBUG from even potentially
> contributing. So I'll re-run this test.
> 
> For now I presume that what I reported was okay and so
> I continue to refer to these figures later below.

The retry got the same 42 GiByte and 102 GiByte sizes.
(And again I was not monitoring the swap space usage.)

So functionality like ALLOW_OPTIMIZATIONS_FOR_WITH_DEBUG
(keeping optimization flags in CFLAGS) does not contribute
to devel/llvm40's handling. Apparently the CMAKE_BUILD_TYPE
has full control over such and RelWithDebInfo still makes
for a massive build, though not as big as for DEBUG.

For my context I've chosen to go with:

#
# From a local /usr/ports/Mk/bsd.port.mk extension:
ALLOW_OPTIMIZATIONS_FOR_WITH_DEBUG=
#
.if ${.CURDIR:M*/devel/llvm*}
#WITH_DEBUG=
.else
WITH_DEBUG=
.endif
WITH_DEBUG_FILES=

(where ALLOW_OPTIMIZATIONS_FOR_WITH_DEBUG is from a
local change to /usr/ports/Mk/bsd.port.mk but is
not justified via devel/llvm* ports but via behavior
for most other ports.)

>> # svnlite diff /usr/ports/devel/llvm40/
>> Index: /usr/ports/devel/llvm40/Makefile
>> ===================================================================
>> --- /usr/ports/devel/llvm40/Makefile	(revision 436747)
>> +++ /usr/ports/devel/llvm40/Makefile	(working copy)
>> _at__at_ -236,6 +236,11 _at__at_
>> 
>> .include <bsd.port.pre.mk>
>> 
>> +.if defined(WITH_DEBUG)
>> +CMAKE_BUILD_TYPE=	RelWithDebInfo
>> +STRIP=
>> +.endif
>> +
>> _CRTLIBDIR=	${LLVM_PREFIX:S|${PREFIX}/||}/lib/clang/${LLVM_RELEASE}/lib/freebsd
>> .if ${ARCH} == "amd64"
>> _COMPILER_RT_LIBS= \
>> 
>> 
>> 
>> pkg delete after the build reports:
>> 
>> Installed packages to be REMOVED:
>> 	llvm40-4.0.0
>> 
>> Number of packages to be removed: 1
>> 
>> The operation will free 42 GiB.
>> 
>> So down by 7 GiBytes from 49 GiBytes.
>> 
>> (I did not actually delete it.)
>> 
>> Also:
>> 
>> # du -sg /usr/obj/portswork/usr/ports/devel/llvm40
>> 102	/usr/obj/portswork/usr/ports/devel/llvm40
>> 
>> which is down by 16 GiBytes from 118 GiBytes.
>> 
>> Reminder: These are from portmaster -DK so no
>> cleanup after the build, which is what leaves
>> the source code and such around in case of
>> needing to look at a problem.
>> 
>> (102+42) GiBytes == 146 GiBytes.
>> vs.
>> (118+49) GiBytes == 167 GiBytes.
>> 
>> So a difference of 21 GiBytes (or so).
>> 
>> But that is for everything in each case (and
>> WITH_DEBUG= in use):
>> 
>> # more /var/db/ports/devel_llvm40/options
>> # This file is auto-generated by 'make config'.
>> # Options for llvm40-4.0.0.r4
>> _OPTIONS_READ=llvm40-4.0.0.r4
>> _FILE_COMPLETE_OPTIONS_LIST=CLANG DOCS EXTRAS LIT LLD LLDB
>> OPTIONS_FILE_SET+=CLANG
>> OPTIONS_FILE_SET+=DOCS
>> OPTIONS_FILE_SET+=EXTRAS
>> OPTIONS_FILE_SET+=LIT
>> OPTIONS_FILE_SET+=LLD
>> OPTIONS_FILE_SET+=LLDB
>> 
>> So avoiding WITH_DEBUG= and/or various build options
>> is still the major way of avoiding use of lots of space
>> if it is an issue.
>> 
>> 
>> 
>> Why no RAM+SWAP total report this time:
>> 
>> As far as I know FreeBSD does not track or report peak
>> swap-space usage since the last boot. And, unfortunately
>> I was not around to just sit and watch a top display this
>> time and I did not set up any periodic recording into a
>> file.
>> 
>> That is why I've not reported on the RAM+SWAP total
>> this time. It will have to be another experiment
>> some other time.
>> 
>> [I do wish FreeBSD had a way of reporting peak swap-space
>> usage.]
> 
> I've also tried without WITH_DEBUG= and now. . .
> 
> 
> # pkg delete llvm40
> Checking integrity... done (0 conflicting)
> Deinstallation has been requested for the following 1 packages (of 0 packages in the universe):
> 
> Installed packages to be REMOVED:
> 	llvm40-4.0.0
> 
> Number of packages to be removed: 1
> 
> The operation will free 1 GiB.
> 
> Proceed with deinstalling packages? [y/N]: n
> 
> 
> # du -sg /usr/obj/portswork/usr/ports/devel/llvm40/
> 5	/usr/obj/portswork/usr/ports/devel/llvm40/
> 
> 
> So the alternatives (with everything built each time):
> 
> (5+1)    GiBytes ==   6 GiBytes. (without WITH_DEBUG=)
> vs.
> (102+42) GiBytes == 146 GiBytes. (WITH_DEBUG= but the adjusted llvm40/Makefiele)
> vs.
> (118+49) GiBytes == 167 GiBytes. (WITH_DEBUG= with Makefile adjustment)
> 
> 
> I'll likely end up having /etc/make.conf contain
> something like the following for most or all of
> my FreeBSD environments:
> 
> #
> # From a local /usr/ports/Mk/bsd.port.mk extension:
> ALLOW_OPTIMIZATIONS_FOR_WITH_DEBUG=
> #
> .if ${.CURDIR:M*/devel/llvm*}
> #WITH_DEBUG=
> .else
> WITH_DEBUG=
> .endif
> WITH_DEBUG_FILES=
> 
> 
> Along with using:
> 
> # svnlite diff /usr/ports/Mk/
> Index: /usr/ports/Mk/bsd.port.mk
> ===================================================================
> --- /usr/ports/Mk/bsd.port.mk	(revision 436747)
> +++ /usr/ports/Mk/bsd.port.mk	(working copy)
> _at__at_ -1646,7 +1646,11 _at__at_
> STRIP_CMD=	${TRUE}
> .endif
> DEBUG_FLAGS?=	-g
> +.if defined(ALLOW_OPTIMIZATIONS_FOR_WITH_DEBUG)
> +CFLAGS:=		${CFLAGS} ${DEBUG_FLAGS}
> +.else
> CFLAGS:=		${CFLAGS:N-O*:N-fno-strict*} ${DEBUG_FLAGS}
> +.endif
> .if defined(INSTALL_TARGET)
> INSTALL_TARGET:=	${INSTALL_TARGET:S/^install-strip$/install/g}
> .endif
> 
> 
> [Although apparently the ALLOW_OPTIMIZATIONS_FOR_WITH_DEBUG
> use has no effect for devel/llvm40 .]

===
Mark Millard
markmi at dsl-only.net
Received on Mon Apr 03 2017 - 16:38:59 UTC

This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:41:11 UTC