Re: Crashes in libthr?

From: Larry Rosenman <ler_at_lerctr.org>
Date: Sun, 13 Mar 2016 21:51:25 -0500
On Sun, Mar 13, 2016 at 07:03:53PM -0500, Larry Rosenman wrote:
> On 2016-03-13 14:29, Konstantin Belousov wrote:
> > On Sun, Mar 13, 2016 at 02:10:58PM -0500, Larry Rosenman wrote:
> >> On 2016-03-13 13:58, Konstantin Belousov wrote:
> >> > On Sun, Mar 13, 2016 at 01:32:20PM -0500, Larry Rosenman wrote:
> >> >> On 2016-03-13 13:12, Konstantin Belousov wrote:
> >> >> > On Sun, Mar 13, 2016 at 11:16:20AM -0500, Larry Rosenman wrote:
> >> >> >> I updated one of my servers, and WHILE DOING THE INSTALLWORLD, I get
> >> >> >> segfaults.
> >> >> >>
> >> >> >> ANY multithreaded program crashes.
> >> >> >>
> >> >> >> I reverted libthr, and it's fine.
> >> >> >>
> >> >> >> borg.lerctr.org / # gdb -c zfs.core /sbin/zfs
> >> >> >> GNU gdb 6.1.1 [FreeBSD]
> >> >> >> Copyright 2004 Free Software Foundation, Inc.
> >> >> >> GDB is free software, covered by the GNU General Public License, and
> >> >> >> you
> >> >> >> are
> >> >> >> welcome to change it and/or distribute copies of it under certain
> >> >> >> conditions.
> >> >> >> Type "show copying" to see the conditions.
> >> >> >> There is absolutely no warranty for GDB.  Type "show warranty" for
> >> >> >> details.
> >> >> >> This GDB was configured as "amd64-marcel-freebsd"...
> >> >> >> Core was generated by `zfs'.
> >> >> >> Program terminated with signal 11, Segmentation fault.
> >> >> >> Reading symbols from /lib/libjail.so.1...Reading symbols from
> >> >> >> /usr/lib/debug//lib/libjail.so.1.debug...done.
> >> >> >> done.
> >> >> >> Loaded symbols for /lib/libjail.so.1
> >> >> >> Reading symbols from /lib/libnvpair.so.2...Reading symbols from
> >> >> >> /usr/lib/debug//lib/libnvpair.so.2.debug...done.
> >> >> >> done.
> >> >> >> Loaded symbols for /lib/libnvpair.so.2
> >> >> >> Reading symbols from /lib/libuutil.so.2...Reading symbols from
> >> >> >> /usr/lib/debug//lib/libuutil.so.2.debug...done.
> >> >> >> done.
> >> >> >> Loaded symbols for /lib/libuutil.so.2
> >> >> >> Reading symbols from /lib/libzfs_core.so.2...Reading symbols from
> >> >> >> /usr/lib/debug//lib/libzfs_core.so.2.debug...done.
> >> >> >> done.
> >> >> >> Loaded symbols for /lib/libzfs_core.so.2
> >> >> >> Reading symbols from /lib/libzfs.so.2...Reading symbols from
> >> >> >> /usr/lib/debug//lib/libzfs.so.2.debug...done.
> >> >> >> done.
> >> >> >> Loaded symbols for /lib/libzfs.so.2
> >> >> >> Reading symbols from /lib/libc.so.7...Reading symbols from
> >> >> >> /usr/lib/debug//lib/libc.so.7.debug...done.
> >> >> >> done.
> >> >> >> Loaded symbols for /lib/libc.so.7
> >> >> >> Reading symbols from /lib/libmd.so.6...Reading symbols from
> >> >> >> /usr/lib/debug//lib/libmd.so.6.debug...done.
> >> >> >> done.
> >> >> >> Loaded symbols for /lib/libmd.so.6
> >> >> >> Reading symbols from /lib/libumem.so.2...Reading symbols from
> >> >> >> /usr/lib/debug//lib/libumem.so.2.debug...done.
> >> >> >> done.
> >> >> >> Loaded symbols for /lib/libumem.so.2
> >> >> >> Reading symbols from /lib/libutil.so.9...Reading symbols from
> >> >> >> /usr/lib/debug//lib/libutil.so.9.debug...done.
> >> >> >> done.
> >> >> >> Loaded symbols for /lib/libutil.so.9
> >> >> >> Reading symbols from /lib/libm.so.5...Reading symbols from
> >> >> >> /usr/lib/debug//lib/libm.so.5.debug...done.
> >> >> >> done.
> >> >> >> Loaded symbols for /lib/libm.so.5
> >> >> >> Reading symbols from /lib/libavl.so.2...Reading symbols from
> >> >> >> /usr/lib/debug//lib/libavl.so.2.debug...done.
> >> >> >> done.
> >> >> >> Loaded symbols for /lib/libavl.so.2
> >> >> >> Reading symbols from /lib/libbsdxml.so.4...Reading symbols from
> >> >> >> /usr/lib/debug//lib/libbsdxml.so.4.debug...done.
> >> >> >> done.
> >> >> >> Loaded symbols for /lib/libbsdxml.so.4
> >> >> >> Reading symbols from /lib/libgeom.so.5...Reading symbols from
> >> >> >> /usr/lib/debug//lib/libgeom.so.5.debug...done.
> >> >> >> done.
> >> >> >> Loaded symbols for /lib/libgeom.so.5
> >> >> >> Reading symbols from /lib/libz.so.6...Reading symbols from
> >> >> >> /usr/lib/debug//lib/libz.so.6.debug...done.
> >> >> >> done.
> >> >> >> Loaded symbols for /lib/libz.so.6
> >> >> >> Reading symbols from /lib/libthr.so.3...done.
> >> >> >> Loaded symbols for /lib/libthr.so.3
> >> >> > Why all libs have debug symbols, while your most interesting one,
> >> >> > libthr.so.3, does not ?
> >> >> >
> >> >> >> Reading symbols from /lib/libsbuf.so.6...Reading symbols from
> >> >> >> /usr/lib/debug//lib/libsbuf.so.6.debug...done.
> >> >> >> done.
> >> >> >> Loaded symbols for /lib/libsbuf.so.6
> >> >> >> Reading symbols from /libexec/ld-elf.so.1...done.
> >> >> >> Loaded symbols for /libexec/ld-elf.so.1
> >> >> >> #0  0x0000000802703f81 in __pthread_cxa_finalize () from
> >> >> >> /lib/libthr.so.3
> >> >> >> [New LWP 100957]
> >> >> >> (gdb) bt
> >> >> >> #0  0x0000000802703f81 in __pthread_cxa_finalize () from
> >> >> >> /lib/libthr.so.3
> >> >> >> #1  0x0000000802703e85 in __pthread_cxa_finalize () from
> >> >> >> /lib/libthr.so.3
> >> >> >> #2  0x0000000802707052 in ?? () from /lib/libthr.so.3
> >> >> >> #3  0x000000080063fc00 in ?? ()
> >> >> >> #4  0x00007fffffffe638 in ?? ()
> >> >> >> #5  0x00007fffffffe5b0 in ?? ()
> >> >> >> #6  0x00000008026f8fd6 in atoi_at_plt () from /lib/libthr.so.3
> >> >> >> #7  0x00007fffffffe5b0 in ?? ()
> >> >> >> #8  0x000000080061adfd in r_debug_state () from /libexec/ld-elf.so.1
> >> >> >> Previous frame inner to this frame (corrupt stack?)
> >> >> >> (gdb)
> >> >> >>
> >> >> >> old SVN: r296103
> >> >> >> new SVN: r296796M
> >> >> >> (The M is a nd6 patch from markj_at_)
> >> >> >>
> >> >> >> this was a FULL buildworld/buildkernel.
> >> >> >
> >> >> > If you cd lib/libthr and do
> >> >> > 	make clean all install DEBUG_FLAGS=-g
> >> >> > on the broken world, does it fix the problem ?  If not, do debugging
> >> >> > symbols from libthr appear accessible to gdb at least ?  Try this to
> >> >> > get useful backtrace with source lines information.
> >> >> ar crashes linking the library......
> >> >
> >> > ar does not link library, it archives .o into .a archive.
> >> > That said, neither ar nor ld (I am not sure whether 'ar' or 'linking'
> >> > is a thinko in the sentence above) do not depend on libthr. You have
> >> > something more fundamental broken.
> >> >
> >> > I put the libthr.so.3 built with the debugging symbols on amd64, using
> >> > head
> >> > r296779, at https://people.freebsd.org/~kib/misc/libthr.so.3 .  Try
> >> > with
> >> > it by manually copying the file to /lib.  If still crashing, it should
> >> > show the line numbers.
> >> no dice.
> >> 
> >> (gdb) borg.lerctr.org /home/ler $ sudo gdb -c ar.core /usr/bin/ar
> >> Password:
> >> GNU gdb 6.1.1 [FreeBSD]
> >> Copyright 2004 Free Software Foundation, Inc.
> >> GDB is free software, covered by the GNU General Public License, and 
> >> you
> >> are
> >> welcome to change it and/or distribute copies of it under certain
> >> conditions.
> >> Type "show copying" to see the conditions.
> >> There is absolutely no warranty for GDB.  Type "show warranty" for
> >> details.
> >> This GDB was configured as "amd64-marcel-freebsd"...
> >> Core was generated by `ar'.
> >> Program terminated with signal 11, Segmentation fault.
> >> #0  0x000000000042c271 in _thr_rtld_init ()
> >> [New Thread 800a1b000 (LWP 100317/<unknown>)]
> >> (gdb) bt
> >> #0  0x000000000042c271 in _thr_rtld_init ()
> >> #1  0x000000000042c19a in _libpthread_init ()
> >> #2  0x00000000004eaa82 in __do_global_ctors_aux ()
> >> #3  0x0000000000400196 in _init ()
> >> #4  0x00007fffffffebd0 in ?? ()
> >> #5  0x00000000004002a6 in _start ()
> >> #6  0x0000000000000000 in ?? ()
> >> (gdb)
> >> 
> > 
> > So your ar is linked to libthr, ok.  But since the binary is statically
> > linked, of course putting libthr.so.3 in place would not add symbols
> > for ar.
> > 
> > Try to run some dynamically linked binary which uses libthr.  E.g. both
> > ntpq and zfs qualify.
> zfs works with your libthr.
> 
> I'm doing a full no -j buildworld and we'll see where we get.  I used 
> the ar from a zfs snap to get buildworld to run.
> 
> 
I wound up restoring EVERYTHING from the latest snapshot memstick image
and I still can't get a world built:

attached is the make.out, from a clean /usr/obj, and latest /usr/src.


-- 
Larry Rosenman                     http://www.lerctr.org/~ler
Phone: +1 214-642-9640                 E-Mail: ler_at_lerctr.org
US Mail: 7011 W Parmer Ln, Apt 1115, Austin, TX 78729-6961

Received on Mon Mar 14 2016 - 01:51:39 UTC

This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:41:03 UTC