Re: Crashes in libthr?

On Sun, Mar 13, 2016 at 11:16:20AM -0500, Larry Rosenman wrote:
> I updated one of my servers, and WHILE DOING THE INSTALLWORLD, I get 
> segfaults.
> 
> ANY multithreaded program crashes.
> 
> I reverted libthr, and it's fine.
> 
> borg.lerctr.org / # gdb -c zfs.core /sbin/zfs
> GNU gdb 6.1.1 [FreeBSD]
> Copyright 2004 Free Software Foundation, Inc.
> GDB is free software, covered by the GNU General Public License, and you 
> are
> welcome to change it and/or distribute copies of it under certain 
> conditions.
> Type "show copying" to see the conditions.
> There is absolutely no warranty for GDB.  Type "show warranty" for 
> details.
> This GDB was configured as "amd64-marcel-freebsd"...
> Core was generated by `zfs'.
> Program terminated with signal 11, Segmentation fault.
> Reading symbols from /lib/libjail.so.1...Reading symbols from 
> /usr/lib/debug//lib/libjail.so.1.debug...done.
> done.
> Loaded symbols for /lib/libjail.so.1
> Reading symbols from /lib/libnvpair.so.2...Reading symbols from 
> /usr/lib/debug//lib/libnvpair.so.2.debug...done.
> done.
> Loaded symbols for /lib/libnvpair.so.2
> Reading symbols from /lib/libuutil.so.2...Reading symbols from 
> /usr/lib/debug//lib/libuutil.so.2.debug...done.
> done.
> Loaded symbols for /lib/libuutil.so.2
> Reading symbols from /lib/libzfs_core.so.2...Reading symbols from 
> /usr/lib/debug//lib/libzfs_core.so.2.debug...done.
> done.
> Loaded symbols for /lib/libzfs_core.so.2
> Reading symbols from /lib/libzfs.so.2...Reading symbols from 
> /usr/lib/debug//lib/libzfs.so.2.debug...done.
> done.
> Loaded symbols for /lib/libzfs.so.2
> Reading symbols from /lib/libc.so.7...Reading symbols from 
> /usr/lib/debug//lib/libc.so.7.debug...done.
> done.
> Loaded symbols for /lib/libc.so.7
> Reading symbols from /lib/libmd.so.6...Reading symbols from 
> /usr/lib/debug//lib/libmd.so.6.debug...done.
> done.
> Loaded symbols for /lib/libmd.so.6
> Reading symbols from /lib/libumem.so.2...Reading symbols from 
> /usr/lib/debug//lib/libumem.so.2.debug...done.
> done.
> Loaded symbols for /lib/libumem.so.2
> Reading symbols from /lib/libutil.so.9...Reading symbols from 
> /usr/lib/debug//lib/libutil.so.9.debug...done.
> done.
> Loaded symbols for /lib/libutil.so.9
> Reading symbols from /lib/libm.so.5...Reading symbols from 
> /usr/lib/debug//lib/libm.so.5.debug...done.
> done.
> Loaded symbols for /lib/libm.so.5
> Reading symbols from /lib/libavl.so.2...Reading symbols from 
> /usr/lib/debug//lib/libavl.so.2.debug...done.
> done.
> Loaded symbols for /lib/libavl.so.2
> Reading symbols from /lib/libbsdxml.so.4...Reading symbols from 
> /usr/lib/debug//lib/libbsdxml.so.4.debug...done.
> done.
> Loaded symbols for /lib/libbsdxml.so.4
> Reading symbols from /lib/libgeom.so.5...Reading symbols from 
> /usr/lib/debug//lib/libgeom.so.5.debug...done.
> done.
> Loaded symbols for /lib/libgeom.so.5
> Reading symbols from /lib/libz.so.6...Reading symbols from 
> /usr/lib/debug//lib/libz.so.6.debug...done.
> done.
> Loaded symbols for /lib/libz.so.6
> Reading symbols from /lib/libthr.so.3...done.
> Loaded symbols for /lib/libthr.so.3
Why all libs have debug symbols, while your most interesting one,
libthr.so.3, does not ?

> Reading symbols from /lib/libsbuf.so.6...Reading symbols from 
> /usr/lib/debug//lib/libsbuf.so.6.debug...done.
> done.
> Loaded symbols for /lib/libsbuf.so.6
> Reading symbols from /libexec/ld-elf.so.1...done.
> Loaded symbols for /libexec/ld-elf.so.1
> #0  0x0000000802703f81 in __pthread_cxa_finalize () from 
> /lib/libthr.so.3
> [New LWP 100957]
> (gdb) bt
> #0  0x0000000802703f81 in __pthread_cxa_finalize () from 
> /lib/libthr.so.3
> #1  0x0000000802703e85 in __pthread_cxa_finalize () from 
> /lib/libthr.so.3
> #2  0x0000000802707052 in ?? () from /lib/libthr.so.3
> #3  0x000000080063fc00 in ?? ()
> #4  0x00007fffffffe638 in ?? ()
> #5  0x00007fffffffe5b0 in ?? ()
> #6  0x00000008026f8fd6 in atoi_at_plt () from /lib/libthr.so.3
> #7  0x00007fffffffe5b0 in ?? ()
> #8  0x000000080061adfd in r_debug_state () from /libexec/ld-elf.so.1
> Previous frame inner to this frame (corrupt stack?)
> (gdb)
> 
> old SVN: r296103
> new SVN: r296796M
> (The M is a nd6 patch from markj_at_)
> 
> this was a FULL buildworld/buildkernel.

If you cd lib/libthr and do
	make clean all install DEBUG_FLAGS=-g
on the broken world, does it fix the problem ?  If not, do debugging
symbols from libthr appear accessible to gdb at least ?  Try this to
get useful backtrace with source lines information.