Re: FreeBSD deadlock (with fork?)

On Thursday 18 September 2008 19:13:19 Daniel Eischen wrote:
> On Thu, 18 Sep 2008, David Naylor wrote:
> > On Thursday 18 September 2008 15:33:52 Tom Evans wrote:
> >> On Thu, 2008-09-18 at 06:31 +0200, David Naylor wrote:
> >>> Hi,
> >>>
> >>> I have a program that spawns a lot of subprocesses (with pipes open)
> >>> from multiple threads.  The problem is the program often deadlocks, but
> >>> not consistently.  Sometimes the program can run over 5 times to
> >>> competition without incidence and yet othertimes it locks within a few
> >>> seconds.
> >>
> >> Do you create threads, which then fork(), or do you fork() and then
> >> create threads?
> >
> > I have many threads that then fork.  (aka forking threads).
> >
> >> I think the former will not work..
> >
> > Is there any reason for this and is this a FreeBSD limitation or a
> > general problem?
>
> No, the former will work.  You can fork() from threads, only as long
> as your forked processes only call async-signal-safe functions or
> some form of exec().  For instance, you can't fork and then create
> new threads from that child process.  If you are not immediately
> exec()'ing from the child process, then you have to be careful
> and only use async-signal-safe functions.
>
> Remember that you have multiple threads in the parent, so the
> state of libthr, libc, etc may be inconsistent in a child
> process.

If you have a look at the backtrace from my original post you will see that 
all my threads are getting locked.  The program I am using is python and the 
call that is forking is Popen(['some', 'program'], stdout=PIPE, 
stderr=STDOUT).  As far as I know it just sets up the pipes and does an 
execvp.  (See 
http://svn.python.org/view/*checkout*/python/tags/r252/Lib/subprocess.py?content-type=text%2Fplain&rev=60915 
from line 981 to 1091).  [[I tried appending close_fds=True to the argument, 
it appeared to make the program deadlock less often...]]

Any ideas as to what might be causing this dead lock (and why all the threads 
are stopping in amd64 specific assembler)?