Re: panic: Exit: Single threading fouled up

On Mon, 26 Apr 2004, Dan Nelson wrote:

> In the last episode (Apr 26), Gavin Atkinson said:
> > I've seen this panic twice now, once on a heavily loaded UP machine
> > running gnome at the time, and once on an SMP (hyperthreaded) machine
> > which was mostly idle as it was shutting down. Both running with ULE.
> 
> I've gotten it 6 times while running the pike testsuite, but not
> reliably enough that I can run a WITNESS kernel for a couple hours and
> catch it.  SMP system, 4BSD scheduler, libpthread.  Hangs trying to
> flush buffers so it has never generated a crashdump.  The couple of
> times I was able to break into the debugger before the hang, a ps
> showed most of the processes in the system waiting for the "proctree"
> mutex.

By chance I'm reading around that code at the moment..
here's what is going on..

When a threaded process exits, all threads except the one that is
actually doing the exit() are forced to abort.
THEORETICALLY the first thread to get to this code
should run thread_single() and if another thread calls exit() it should
block looking for the proc lock until the first thread has successfully
set the "die-you-scum" flag and then proceed on and see that flag and
just commit suicide.

actual code is (simplified):

        PROC_LOCK(p);
        if (p->p_flag & P_SA || p->p_numthreads > 1) {
                thread_suspend_check(0);
                if (thread_single(SINGLE_EXIT))
                        panic ("Exit: Single threading fouled up");
                /*
where
thread_single(SINGLE_EXIT)
is: (simplified)

thread_single(int force_exit)
{
        struct proc *p;

        td = curthread;
        p = td->td_proc;
          
        if ((p->p_flag & P_SA) == 0 && p->p_numthreads == 1)
                return (0);

        /* Is someone already single threading? */
        if (p->p_singlethread)
                return (1);

        if (force_exit == SINGLE_EXIT) {
                p->p_flag |= P_SINGLE_EXIT;
        } else
                p->p_flag &= ~P_SINGLE_EXIT;
        p->p_flag |= P_STOPPED_SINGLE;
        mtx_lock_spin(&sched_lock);
        p->p_singlethread = td;
[...]
	[set flags that should trigger thread_suspend_check()]
	return (0);
}

This means that, despite the fact that the proc lock is required,
two threads have managed to get into the same code.

the thread_suspend_check(0); for the 2nd thread coming in should just
abort the thread and never return, so it should never proceed on to see
the p->p_singlethread already set to non-NULL.

but it does.. hence this panic..

possibly something is setting it earlier 
(There are other times we force single-threading)
and forgetting to unset it.
(e.g. in fork or exec) 

it is unset using the thread_single_end() call.



> 
> My trace:
> 
> panic: Exit: Single threading fouled up
> at line 157 in file ../../../kern/kern_exit.c
> cpuid = 1; 
> Stack backtrace:
> __panic(c075f5cb,9d,c075f611,86,e0589c80) at __panic+0x1a2
> exit1(c54d3690,9,c0761f54,971,1) at exit1+0x10e5
> sigexit(c54d3690,9,c0761f54,8fd,c4a27aa8) at sigexit+0xff
> postsig(9,8,c0764ca5,100,c075f6a6) at postsig+0x32d
> ast(e0589d48) at ast+0x2f4
> doreti_ast() at doreti_ast+0x17
> boot() called on cpu#1
>  
> -- 
> 	Dan Nelson
> 	dnelson_at_allantgroup.com
> _______________________________________________
> freebsd-current_at_freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-current
> To unsubscribe, send any mail to "freebsd-current-unsubscribe_at_freebsd.org"
>