Re: Deterministic rescue buildworld error with custom make.conf/src.conf/MAKEOBJDIRPREFIX

On Sun, 2017-03-12 at 13:27 +1100, Lawrence Stewart wrote:
> Hi Ian,
> 
> On 12/03/2017 10:29, Ian Lepore wrote:
> > 
> > On Sun, 2017-03-12 at 10:22 +1100, Lawrence Stewart wrote:
> > > 
> > > Hi all,
> > > 
> > > I'm unable to complete buildworld with 2 recent svn revs I've
> > > tried
> > > (r314838 and r315059). I'm building for a slightly resource
> > > constrained
> > > production system so am specifying custom settings and a
> > > different
> > > obj
> > > tree location so I can copy it to the target system. The error
> > > persists
> > > after an "rm -rf /usr/obj/*", and if parallel building is
> > > disabled.
> > > 
> > > The underlying build system built from r314838 via simple "make
> > > -C
> > > /usr/src -s -j6 buildworld buildkernel" built and installed fine,
> > > so
> > > the
> > > problem seems to be around the use of the build customisations.
> > > 
> > > Any clues?
> > > 
> > > Cheers,
> > > Lawrence
> > > 
> > > 
> > > root_at_builder-head-amd64:/usr/src # cat cust_make.conf
> > > KERNCONF=GENERIC-NODEBUG
> > > MALLOC_PRODUCTION=YES
> > > 
> > > root_at_builder-head-amd64:/usr/src # cat cust_src.conf
> > > WITHOUT_PROFILE=1
> > > 
> > > root_at_builder-head-amd64:/usr/src # make
> > > __MAKE_CONF=/usr/src/cust_make.conf
> > > SRCCONF=/usr/src/cust_src.conf
> > > MAKEOBJDIRPREFIX=/usr/obj/cust buildworld buildkernel
> > > [...]
> > > MK_AUTO_OBJ=no
> > > MK_TESTS=no  UPDATE_DEPENDFILE=no  _RECURSING_CRUNCH=1
> > > CC="cc -target x86_64-unknown-freebsd12.0
> > > --sysroot=/usr/obj/cust/usr/src/tmp
> > > -B/usr/obj/cust/usr/src/tmp/usr/bin
> > > -O2 -pipe   -std=gnu99    -Qunused-arguments  "  CXX="c++  -
> > > target
> > > x86_64-unknown-freebsd12.0 --sysroot=/usr/obj/cust/usr/src/tmp
> > > -B/usr/obj/cust/usr/src/tmp/usr/bin -O2 -pipe -Qunused-arguments
> > > -Wno-c++11-extensions  "  make .MAKE.MODE="normal curdirOk=yes"
> > > .MAKE.META.IGNORE_PATHS=""  -f rescue.mk exe
> > > cc -target x86_64-unknown-freebsd12.0
> > > --sysroot=/usr/obj/cust/usr/src/tmp
> > > -B/usr/obj/cust/usr/src/tmp/usr/bin
> > > -O2 -pipe   -std=gnu99    -Qunused-arguments   -nostdlib -Wl,-dc
> > > -r
> > > -o
> > > cat.lo cat_stub.o
> > > /usr/obj/cust/usr/src/rescue/rescue//usr/src/bin/cat/cat.o
> > > cc: error: no such file or directory:
> > > '/usr/obj/cust/usr/src/rescue/rescue//usr/src/bin/cat/cat.o'
> > > *** Error code 1
> > > 
> > > There appear to be a lot of missing .o files under the rescue obj
> > > tree:
> > > 
> > > root_at_builder-head-amd64:/usr/src # find
> > > /usr/obj/cust/usr/src/rescue/rescue//usr -type f
> > > /usr/obj/cust/usr/src/rescue/rescue//usr/src/bin/sh/mksyntax.o
> > > /usr/obj/cust/usr/src/rescue/rescue//usr/src/bin/sh/mksyntax
> > > /usr/obj/cust/usr/src/rescue/rescue//usr/src/bin/sh/mknodes.o
> > > /usr/obj/cust/usr/src/rescue/rescue//usr/src/bin/sh/mknodes
> > > /usr/obj/cust/usr/src/rescue/rescue//usr/src/bin/csh/sh.err.h
> > > /usr/obj/cust/usr/src/rescue/rescue//usr/src/bin/csh/tc.const.h
> > > /usr/obj/cust/usr/src/rescue/rescue//usr/src/bin/csh/gethost
> > > 
> > > compared with an obj tree on a different head system:
> > > 
> > > find /usr/obj/usr/src/rescue/rescue/usr/ -type f | wc -l
> > >     1552
> > > _______________________________________________
> > > freebsd-current_at_freebsd.org mailing list
> > > https://lists.freebsd.org/mailman/listinfo/freebsd-current
> > > To unsubscribe, send any mail to "freebsd-current-unsubscribe_at_fre
> > > ebsd
> > > .org"
> > The MAKEOBJDIRPREFIX variable must be set in the environment, not
> > in
> > make.conf or on the make command line (documented in build(7)).
> Your assertion seems at odds with my past experience and my reading
> of
> the man page... from build(7):
> 
> 	The build may be controlled by defining make(1) variables
> 	described in the ENVIRONMENT section below, and by the
> 	variables documented in make.conf(5).
> 
> ... which indicates they are make variables, not environment
> variables
> specifically. As a concrete example, TARGET and DESTDIR are listed
> under
> the "ENVIRONMENT" section of the man page, yet "EXAMPLES" shows:
> 
>            make TARGET=sparc64 buildworld
>            make TARGET=sparc64 DESTDIR=/clients/sparc64 installworld
> 
> I've certainly always set build vars documented in the "ENVIRONMENT"
> section of the man page on the make command line without issue.
> Pretty
> sure I've set MAKEOBJDIRPREFIX from the make command line also in the
> past, though perhaps it has been working for me "by accident" and a
> documentation tweak is in order if the distinction you make is in
> fact
> relevant...
> 
> Cheers,
> Lawrence
> _______________________________________________
> freebsd-current_at_freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-current
> To unsubscribe, send any mail to "freebsd-current-unsubscribe_at_freebsd
> .org"

You cited everything from build(7) except the part most on-point for
the problem you're having:

    MAKEOBJDIRPREFIX
        Defines the prefix for directory names in the tree of
        built objects.  Defaults to /usr/obj if not defined.  This variable
        should only be set in the environment and not via /etc/make.conf or the
        command line.

-- Ian