Re: Crashes in libthr?

On Sun, Mar 13, 2016 at 02:10:58PM -0500, Larry Rosenman wrote:
> On 2016-03-13 13:58, Konstantin Belousov wrote:
> > On Sun, Mar 13, 2016 at 01:32:20PM -0500, Larry Rosenman wrote:
> >> On 2016-03-13 13:12, Konstantin Belousov wrote:
> >> > On Sun, Mar 13, 2016 at 11:16:20AM -0500, Larry Rosenman wrote:
> >> >> I updated one of my servers, and WHILE DOING THE INSTALLWORLD, I get
> >> >> segfaults.
> >> >>
> >> >> ANY multithreaded program crashes.
> >> >>
> >> >> I reverted libthr, and it's fine.
> >> >>
> >> >> borg.lerctr.org / # gdb -c zfs.core /sbin/zfs
> >> >> GNU gdb 6.1.1 [FreeBSD]
> >> >> Copyright 2004 Free Software Foundation, Inc.
> >> >> GDB is free software, covered by the GNU General Public License, and
> >> >> you
> >> >> are
> >> >> welcome to change it and/or distribute copies of it under certain
> >> >> conditions.
> >> >> Type "show copying" to see the conditions.
> >> >> There is absolutely no warranty for GDB.  Type "show warranty" for
> >> >> details.
> >> >> This GDB was configured as "amd64-marcel-freebsd"...
> >> >> Core was generated by `zfs'.
> >> >> Program terminated with signal 11, Segmentation fault.
> >> >> Reading symbols from /lib/libjail.so.1...Reading symbols from
> >> >> /usr/lib/debug//lib/libjail.so.1.debug...done.
> >> >> done.
> >> >> Loaded symbols for /lib/libjail.so.1
> >> >> Reading symbols from /lib/libnvpair.so.2...Reading symbols from
> >> >> /usr/lib/debug//lib/libnvpair.so.2.debug...done.
> >> >> done.
> >> >> Loaded symbols for /lib/libnvpair.so.2
> >> >> Reading symbols from /lib/libuutil.so.2...Reading symbols from
> >> >> /usr/lib/debug//lib/libuutil.so.2.debug...done.
> >> >> done.
> >> >> Loaded symbols for /lib/libuutil.so.2
> >> >> Reading symbols from /lib/libzfs_core.so.2...Reading symbols from
> >> >> /usr/lib/debug//lib/libzfs_core.so.2.debug...done.
> >> >> done.
> >> >> Loaded symbols for /lib/libzfs_core.so.2
> >> >> Reading symbols from /lib/libzfs.so.2...Reading symbols from
> >> >> /usr/lib/debug//lib/libzfs.so.2.debug...done.
> >> >> done.
> >> >> Loaded symbols for /lib/libzfs.so.2
> >> >> Reading symbols from /lib/libc.so.7...Reading symbols from
> >> >> /usr/lib/debug//lib/libc.so.7.debug...done.
> >> >> done.
> >> >> Loaded symbols for /lib/libc.so.7
> >> >> Reading symbols from /lib/libmd.so.6...Reading symbols from
> >> >> /usr/lib/debug//lib/libmd.so.6.debug...done.
> >> >> done.
> >> >> Loaded symbols for /lib/libmd.so.6
> >> >> Reading symbols from /lib/libumem.so.2...Reading symbols from
> >> >> /usr/lib/debug//lib/libumem.so.2.debug...done.
> >> >> done.
> >> >> Loaded symbols for /lib/libumem.so.2
> >> >> Reading symbols from /lib/libutil.so.9...Reading symbols from
> >> >> /usr/lib/debug//lib/libutil.so.9.debug...done.
> >> >> done.
> >> >> Loaded symbols for /lib/libutil.so.9
> >> >> Reading symbols from /lib/libm.so.5...Reading symbols from
> >> >> /usr/lib/debug//lib/libm.so.5.debug...done.
> >> >> done.
> >> >> Loaded symbols for /lib/libm.so.5
> >> >> Reading symbols from /lib/libavl.so.2...Reading symbols from
> >> >> /usr/lib/debug//lib/libavl.so.2.debug...done.
> >> >> done.
> >> >> Loaded symbols for /lib/libavl.so.2
> >> >> Reading symbols from /lib/libbsdxml.so.4...Reading symbols from
> >> >> /usr/lib/debug//lib/libbsdxml.so.4.debug...done.
> >> >> done.
> >> >> Loaded symbols for /lib/libbsdxml.so.4
> >> >> Reading symbols from /lib/libgeom.so.5...Reading symbols from
> >> >> /usr/lib/debug//lib/libgeom.so.5.debug...done.
> >> >> done.
> >> >> Loaded symbols for /lib/libgeom.so.5
> >> >> Reading symbols from /lib/libz.so.6...Reading symbols from
> >> >> /usr/lib/debug//lib/libz.so.6.debug...done.
> >> >> done.
> >> >> Loaded symbols for /lib/libz.so.6
> >> >> Reading symbols from /lib/libthr.so.3...done.
> >> >> Loaded symbols for /lib/libthr.so.3
> >> > Why all libs have debug symbols, while your most interesting one,
> >> > libthr.so.3, does not ?
> >> >
> >> >> Reading symbols from /lib/libsbuf.so.6...Reading symbols from
> >> >> /usr/lib/debug//lib/libsbuf.so.6.debug...done.
> >> >> done.
> >> >> Loaded symbols for /lib/libsbuf.so.6
> >> >> Reading symbols from /libexec/ld-elf.so.1...done.
> >> >> Loaded symbols for /libexec/ld-elf.so.1
> >> >> #0  0x0000000802703f81 in __pthread_cxa_finalize () from
> >> >> /lib/libthr.so.3
> >> >> [New LWP 100957]
> >> >> (gdb) bt
> >> >> #0  0x0000000802703f81 in __pthread_cxa_finalize () from
> >> >> /lib/libthr.so.3
> >> >> #1  0x0000000802703e85 in __pthread_cxa_finalize () from
> >> >> /lib/libthr.so.3
> >> >> #2  0x0000000802707052 in ?? () from /lib/libthr.so.3
> >> >> #3  0x000000080063fc00 in ?? ()
> >> >> #4  0x00007fffffffe638 in ?? ()
> >> >> #5  0x00007fffffffe5b0 in ?? ()
> >> >> #6  0x00000008026f8fd6 in atoi_at_plt () from /lib/libthr.so.3
> >> >> #7  0x00007fffffffe5b0 in ?? ()
> >> >> #8  0x000000080061adfd in r_debug_state () from /libexec/ld-elf.so.1
> >> >> Previous frame inner to this frame (corrupt stack?)
> >> >> (gdb)
> >> >>
> >> >> old SVN: r296103
> >> >> new SVN: r296796M
> >> >> (The M is a nd6 patch from markj_at_)
> >> >>
> >> >> this was a FULL buildworld/buildkernel.
> >> >
> >> > If you cd lib/libthr and do
> >> > 	make clean all install DEBUG_FLAGS=-g
> >> > on the broken world, does it fix the problem ?  If not, do debugging
> >> > symbols from libthr appear accessible to gdb at least ?  Try this to
> >> > get useful backtrace with source lines information.
> >> ar crashes linking the library......
> > 
> > ar does not link library, it archives .o into .a archive.
> > That said, neither ar nor ld (I am not sure whether 'ar' or 'linking'
> > is a thinko in the sentence above) do not depend on libthr. You have
> > something more fundamental broken.
> > 
> > I put the libthr.so.3 built with the debugging symbols on amd64, using 
> > head
> > r296779, at https://people.freebsd.org/~kib/misc/libthr.so.3 .  Try 
> > with
> > it by manually copying the file to /lib.  If still crashing, it should
> > show the line numbers.
> no dice.
> 
> (gdb) borg.lerctr.org /home/ler $ sudo gdb -c ar.core /usr/bin/ar
> Password:
> GNU gdb 6.1.1 [FreeBSD]
> Copyright 2004 Free Software Foundation, Inc.
> GDB is free software, covered by the GNU General Public License, and you 
> are
> welcome to change it and/or distribute copies of it under certain 
> conditions.
> Type "show copying" to see the conditions.
> There is absolutely no warranty for GDB.  Type "show warranty" for 
> details.
> This GDB was configured as "amd64-marcel-freebsd"...
> Core was generated by `ar'.
> Program terminated with signal 11, Segmentation fault.
> #0  0x000000000042c271 in _thr_rtld_init ()
> [New Thread 800a1b000 (LWP 100317/<unknown>)]
> (gdb) bt
> #0  0x000000000042c271 in _thr_rtld_init ()
> #1  0x000000000042c19a in _libpthread_init ()
> #2  0x00000000004eaa82 in __do_global_ctors_aux ()
> #3  0x0000000000400196 in _init ()
> #4  0x00007fffffffebd0 in ?? ()
> #5  0x00000000004002a6 in _start ()
> #6  0x0000000000000000 in ?? ()
> (gdb)
> 

So your ar is linked to libthr, ok.  But since the binary is statically
linked, of course putting libthr.so.3 in place would not add symbols
for ar.

Try to run some dynamically linked binary which uses libthr.  E.g. both
ntpq and zfs qualify.