Re: Fatal error 'mutex is on list' at line 139 in file /usr/src/lib/libthr/thread/thr_mutex.c (errno = 35)

On Fri, Mar 18, 2016 at 07:57:30AM +0200, Oleg V. Nauman wrote:
> On Friday 18 March 2016 03:30:51 Konstantin Belousov wrote:
> > On Thu, Mar 17, 2016 at 09:40:11PM +0200, Oleg V. Nauman wrote:
> > >  I'm experiencing issue with threaded applications ( including many KDE
> > > 
> > > applications ) after recent CURRENT update. Attempt to start application
> > > cause Abort trap (6) with "Fatal error 'mutex is on list' at line 139 in
> > > file /usr/src/lib/libthr/thread/thr_mutex.c (errno = 35)" output on
> > > console.
> > > 
> > > FreeBSD 11.0-CURRENT amd64 r296887
> > > 
> > > Backtrace from application looks like
> > > 
> > > Application: KGpg (kgpg), signal: Abort trap
> > > [Switching to Thread 810815000 (LWP 100865/kgpg)]
> > > [KCrash Handler]
> > > #8  0x0000000807176d6a in thr_kill () from /lib/libc.so.7
> > > #9  0x0000000807176d3b in __raise (s=6) at
> > > /usr/src/lib/libc/gen/raise.c:52
> > > #10 0x0000000807176ca9 in abort () at /usr/src/lib/libc/stdlib/abort.c:65
> > > #11 0x0000000806e9e62c in _thread_exit (fname=<value optimized out>,
> > > lineno=<value optimized out>, msg=<value optimized out>) at
> > > /usr/src/lib/libthr/thread/thr_exit.c:182
> > > #12 0x0000000806e996fc in mutex_lock_common (m=<value optimized out>,
> > > abstime=<value optimized out>, cvattach=<value optimized out>) at
> > > /usr/src/lib/libthr/thread/thr_mutex.c:139
> > > #13 0x0000000806e98c20 in __pthread_mutex_timedlock (mutex=0x814600008,
> > > abstime=0x7fffffff5a58) at /usr/src/lib/libthr/thread/thr_mutex.c:566
> > 
> > From the frame 13, please do
> > p *mutex
> > p *m
> > p **mutex
> 
>  Got few cores today morning, for example:
> 
> gdb /usr/local/bin/akonadi_notes_agent akonadi_notes_agent.core
> .... ( list of many shared libraries loaded by application )
> 
> 0  0x0000000805d08d6a in thr_kill () from /lib/libc.so.7
> [New Thread 817615000 (LWP 100295/<unknown>)]
> (gdb) bt
> #0  0x0000000805d08d6a in thr_kill () from /lib/libc.so.7
> #1  0x0000000805d08d3b in __raise (s=6) at /usr/src/lib/libc/gen/raise.c:52
> #2  0x0000000805d08ca9 in abort () at /usr/src/lib/libc/stdlib/abort.c:65
> #3  0x0000000805a3062c in _thread_exit (fname=<value optimized out>, 
>     lineno=<value optimized out>, msg=<value optimized out>)
>     at /usr/src/lib/libthr/thread/thr_exit.c:182
> #4  0x0000000805a2b6fc in mutex_lock_common (m=<value optimized out>, 
>     abstime=<value optimized out>, cvattach=<value optimized out>)
>     at /usr/src/lib/libthr/thread/thr_mutex.c:139
> #5  0x0000000805a2ac20 in __pthread_mutex_timedlock (mutex=0x81b200008,
>     abstime=0x7fffffffd4f8) at /usr/src/lib/libthr/thread/thr_mutex.c:566
> #6  0x000000080448b560 in KSharedDataCache::setTimestamp ()
>    from /usr/local/lib/libkdecore.so.5
> #7  0x000000080448b7df in KSharedDataCache::setTimestamp ()
>    from /usr/local/lib/libkdecore.so.5
> #8  0x0000000804487192 in KSharedDataCache::insert () from 
> /usr/local/lib/libkdecore.so.5
> #9  0x0000000802c1912e in KIconLoader::drawOverlays () from 
> /usr/local/lib/libkdeui.so.5
> #10 0x0000000802c15e3b in KIconLoader::loadIcon () from 
> /usr/local/lib/libkdeui.so.5
> #11 0x0000000802c14366 in KIconEffect::overlay () from 
> /usr/local/lib/libkdeui.so.5
> #12 0x00000008034aff3d in QIcon::pixmap () from 
> /usr/local/lib/qt4/libQtGui.so.4
> #13 0x000000080349d2d6 in QWidgetPrivate::setWindowIcon_sys ()
>    from /usr/local/lib/qt4/libQtGui.so.4
> #14 0x00000008034590e7 in QWidget::event () from 
> /usr/local/lib/qt4/libQtGui.so.4
> #15 0x000000080340a43c in QApplicationPrivate::notify_helper ()
>    from /usr/local/lib/qt4/libQtGui.so.4
> #16 0x000000080340ca61 in QApplication::notify () from 
> /usr/local/lib/qt4/libQtGui.so.4
> #17 0x0000000802c724b7 in KApplication::notify () from 
> /usr/local/lib/libkdeui.so.5
> #18 0x0000000804d8eaa6 in QCoreApplication::notifyInternal ()
>    from /usr/local/lib/qt4/libQtCore.so.4
> #19 0x00000008034084e1 in QApplication::setWindowIcon () from 
> /usr/local/lib/qt4/libQtGui.so.4
> #20 0x0000000802c74725 in KApplication::iceIOErrorHandler () from 
> /usr/local/lib/libkdeui.so.5
> #21 0x0000000802c72a9b in KApplication::KApplication () from 
> /usr/local/lib/libkdeui.so.5
> #22 0x0000000802c728f7 in KApplication::KApplication () from 
> /usr/local/lib/libkdeui.so.5
> #23 0x000000000040b10c in ?? ()
> #24 0x000000000040a74f in ?? ()
> #25 0x000000080063a000 in ?? ()
> #26 0x0000000000000000 in ?? ()
> (gdb) f 5
> #5  0x0000000805a2ac20 in __pthread_mutex_timedlock (mutex=0x81b200008,
>     abstime=0x7fffffffd4f8) at /usr/src/lib/libthr/thread/thr_mutex.c:566
> 566                     ret = mutex_lock_common(m, abstime, 0);
> Current language:  auto; currently minimal
> (gdb) p *mutex
> $1 = 0x8000000000000001
This means that the mutex is shared, which is very strange.

> (gdb) p *m
> $2 = {m_lock = {m_owner = 1, m_flags = 2147483648, m_ceilings = 0x81b200010,
>     m_spare = 0x81b200018}, m_flags = 0, m_owner = 0, m_count = 0, m_spinloops 
> = 0,
>   m_yieldloops = 0, m_qe = {tqe_next = 0x0, tqe_prev = 0x1}, m_pqe = {
>     tqe_next = 0x71100a00000, tqe_prev = 0x100000000000}}
The values are not reasonable.  At least m_lock.m_owner is wrong, while
m_owner indicates unowned lock.  The m_lock.m_flags is UMUTEX_CONTESTED,
lacking USYNC_PROCESS_SHARED bit.  And of course, m_qe/m_pqe links are
garbage, which triggered the panic.

It almost looks as the *m is gargage memory, but not quite.


> (gdb) p **mutex
> Cannot access memory at address 0x8000000000000001
> 
> 
> > 
> > What non-KDE apps exhibit the behaviour ?
> 
>  I will try to find some simple application experiencing the same issue.

Yes, please.  It would be significantly easier to diagnose the problem if
the minimal example is provided.  If not, please pack one crashing app
and all it non-system libraries and provide the tarball to me.