Re: malloc() call somehow calling the rtld malloc() implementaion

On Fri, Jan 27, 2017 at 12:19:16PM -0500, Alexander Kabaev wrote:
> On Fri, 27 Jan 2017 00:31:30 -0800 (PST)
> Don Lewis <truckman_at_FreeBSD.org> wrote:
> 
> > I've been attempting to get OpenOffice to build properly in a
> > clang400-import poudriere jail and have run into a mystery.  The build
> > procedure creates a c++ executable "idlc", which is used to build
> > other stuff.  The new operator has been overrriden to call a custom
> > memory allocator, which I have configured to call the system version
> > of malloc().
> > 
> > At some point idlc crashes because it has allocated a 16 byte
> > structure and the compiler is using "movaps %xmm0,(%rax)" to
> > initialize it, which requires 16 byte alignment.  Unfortunately this
> > structure is only 8 byte aligned, causing a bus error.  This
> > shouldn't be happening because our system malloc() always seems to do
> > the proper alignment.  It appears that intead of calling the version
> > of malloc() in libc, the simple version of malloc() built into rtld
> > is being called instead.
> > 
> > GNU gdb 6.1.1 [FreeBSD]
> > Copyright 2004 Free Software Foundation, Inc.
> > GDB is free software, covered by the GNU General Public License, and
> > you are welcome to change it and/or distribute copies of it under
> > certain conditions. Type "show copying" to see the conditions.
> > There is absolutely no warranty for GDB.  Type "show warranty" for
> > details. This GDB was configured as "amd64-marcel-freebsd"...
> > (gdb) break main
> > Breakpoint 1 at 0x43b1f6: file idlcmain.cxx, line 34.
> > (gdb) run _at_/tmp/r
> > Starting
> > program: /wrkdirs/usr/ports/editors/openoffice-devel/work/aoo-4.2.0/main/solver/420/unxfbsdx.pro/bin/idlc
> > _at_/tmp/r [New LWP 101304] [New Thread 802616000 (LWP 101304/idlc)]
> > [Switching to Thread 802616000 (LWP 101304/idlc)]
> > 
> > Breakpoint 1, main (argc=2, argv=0x7fffffffb278) at idlcmain.cxx:34
> > 34	SAL_IMPLEMENT_MAIN_WITH_ARGS(argc, argv)
> > (gdb) break malloc
> > Breakpoint 2 at 0x8006a5f01:
> > file /var/poudriere/jails/clang400amd64/usr/src/libexec/rtld-elf/malloc.c,
> > line 163. (gdb) cont Continuing.
> > Trace 12474/1: "Min Prioriy for policy '2' == '0'
> > "
> > Trace 12474/1: "Max Prioriy for policy '2' == '103'
> > "
> > /wrkdirs/usr/ports/editors/openoffice-devel/work/aoo-4.2.0/main/solver/420/unxfbsdx.pro/bin/idlc:
> > compiling 1 source files ...
> > Compiling: /wrkdirs/usr/ports/editors/openoffice-devel/work/aoo-4.2.0/main/offapi/com/sun/star/i18n/KParseTokens.idl
> > 
> > Breakpoint 2, malloc (nbytes=343)
> >     at /var/poudriere/jails/clang400amd64/usr/src/libexec/rtld-elf/malloc.c:163
> > 163	/var/poudriere/jails/clang400amd64/usr/src/libexec/rtld-elf/malloc.c:
> > No such file or directory.
> > in /var/poudriere/jails/clang400amd64/usr/src/libexec/rtld-elf/malloc.c
> > Current language:  auto; currently minimal (gdb) cont
> > Continuing.
> > 
> > Breakpoint 2, malloc (nbytes=32)
> >     at /var/poudriere/jails/clang400amd64/usr/src/libexec/rtld-elf/malloc.c:163
> > 163
> > in /var/poudriere/jails/clang400amd64/usr/src/libexec/rtld-elf/malloc.c
> > (gdb) cont Continuing.
> > Trace 13112/2: "ChildStatusProc : starting
> > '/wrkdirs/usr/ports/editors/openoffice-devel/work/aoo-4.2.0/main/solver/420/unxfbsdx.pro/bin/ucpp'" [New
> > Thread 802616500 (LWP 100249/idlc)] [Switching to Thread 802616500
> > (LWP 100249/idlc)]
> > 
> > Breakpoint 2, malloc (nbytes=19)
> >     at /var/poudriere/jails/clang400amd64/usr/src/libexec/rtld-elf/malloc.c:163
> > 163
> > in /var/poudriere/jails/clang400amd64/usr/src/libexec/rtld-elf/malloc.c
> > (gdb) cont Continuing.
> > 
> > Breakpoint 2, malloc (nbytes=34)
> >     at /var/poudriere/jails/clang400amd64/usr/src/libexec/rtld-elf/malloc.c:163
> > 163
> > in /var/poudriere/jails/clang400amd64/usr/src/libexec/rtld-elf/malloc.c
> > (gdb) cont Continuing.
> > 
> > Breakpoint 2, malloc (nbytes=16)
> >     at /var/poudriere/jails/clang400amd64/usr/src/libexec/rtld-elf/malloc.c:163
> > 163
> > in /var/poudriere/jails/clang400amd64/usr/src/libexec/rtld-elf/malloc.c
> > (gdb) cont Continuing.
> > 
> > Breakpoint 2, malloc (nbytes=16)
> >     at /var/poudriere/jails/clang400amd64/usr/src/libexec/rtld-elf/malloc.c:163
> > 163
> > in /var/poudriere/jails/clang400amd64/usr/src/libexec/rtld-elf/malloc.c
> > (gdb) cont Continuing.
> > 
> > Breakpoint 2, malloc (nbytes=16)
> >     at /var/poudriere/jails/clang400amd64/usr/src/libexec/rtld-elf/malloc.c:163
> > 163
> > in /var/poudriere/jails/clang400amd64/usr/src/libexec/rtld-elf/malloc.c
> > (gdb) cont Continuing.
> > 
> > Breakpoint 2, malloc (nbytes=16)
> >     at /var/poudriere/jails/clang400amd64/usr/src/libexec/rtld-elf/malloc.c:163
> > 163
> > in /var/poudriere/jails/clang400amd64/usr/src/libexec/rtld-elf/malloc.c
> > (gdb) cont Continuing.
> > sizeof(AstExprValue)=16
> > 
> > Program received signal SIGBUS, Bus error.
> > [Switching to Thread 802616000 (LWP 101304/idlc)]
> > 0x0000000000478cc2 in AstExpression::eval_bit_op (this=0x802633dc8, 
> >     ek=EK_const) at astexpression.cxx:1001
> > 1001	    std::auto_ptr< AstExprValue	> retval(new
> > AstExprValue());
> > 
> > 
> > idlc is linked to libc.so:
> > 
> > /wrkdirs/usr/ports/editors/openoffice-devel/work/aoo-4.2.0/main/solver/420/unxfbsdx.pro/bin/idlc:
> > 	libreg.so.3
> > => /wrkdirs/usr/ports/editors/openoffice-devel/work/aoo-4.2.0/main/solver/420/unxfbsdx.pro/lib/libreg.so.3
> > (0x8008b9000) libuno_sal.so.3
> > => /wrkdirs/usr/ports/editors/openoffice-devel/work/aoo-4.2.0/main/solver/420/unxfbsdx.pro/lib/libuno_sal.so.3
> > (0x800c00000) libuno_salhelpergcc3.so.3
> > => /wrkdirs/usr/ports/editors/openoffice-devel/work/aoo-4.2.0/main/solver/420/unxfbsdx.pro/lib/libuno_salhelpergcc3.so.3
> > (0x801040000) libm.so.5 => /lib/libm.so.5 (0x801244000) libc++.so.1
> > => /usr/lib/libc++.so.1 (0x80146e000) libcxxrt.so.1
> > => /lib/libcxxrt.so.1 (0x801735000) libgcc_s.so.1
> > => /lib/libgcc_s.so.1 (0x801953000) libthr.so.3 => /lib/libthr.so.3
> > (0x801b69000) libc.so.7 => /lib/libc.so.7 (0x801d91000) libstore.so.3
> > => /wrkdirs/usr/ports/editors/openoffice-devel/work/aoo-4.2.0/main/solver/420/unxfbsdx.pro/lib/libstore.so.3
> > (0x802152000)
> > 
> > 
> > If I create a simple test program that calls malloc() and set a
> > breakpoint in malloc(), the breakpoint gets set in the rtld version,
> > but the the libc version of malloc is what gets called.
> > 
> > What the heck is going on here, and how can I fix it?
> > 
> 
> rtld on my system does not have malloc exposed as dynamic symbol, it
> cannot possibly be used for symbol resolution by any outside module.

Sure.

Also, the fact that rtld internal malloc is called does not mean that it
is called by the application. There is no backtrace which would describe
the reason for rtld allocating memory in reported cases. Generally,
rtld needs memory for TLS allocation (the program is definitely linked
with libthr, and even libc uses TLS), and, of course, to create data
structures to track dlopen-ed objects.