Hi, I have a program that spawns a lot of subprocesses (with pipes open) from multiple threads. The problem is the program often deadlocks, but not consistently. Sometimes the program can run over 5 times to competition without incidence and yet othertimes it locks within a few seconds. However if I limit the thread count to 1 the problem does not appear to be present. Here are the logs from gdb: (gdb) info thread 5 Thread 7021c0 (LWP 100203) 0x00000008009a2e8c in _umtx_op_err () at /usr/src/lib/libthr/arch/amd64/amd64/_umtx_op_err.S:37 4 Thread a28480 (LWP 100174) 0x00000008009a2e8c in _umtx_op_err () at /usr/src/lib/libthr/arch/amd64/amd64/_umtx_op_err.S:37 3 Thread a61d80 (LWP 100175) 0x00000008009a2e8c in _umtx_op_err () at /usr/src/lib/libthr/arch/amd64/amd64/_umtx_op_err.S:37 2 Thread a61bc0 (LWP 100176) 0x00000008009a2e8c in _umtx_op_err () at /usr/src/lib/libthr/arch/amd64/amd64/_umtx_op_err.S:37 * 1 Thread a61840 (LWP 100177) 0x00000008009a2e8c in _umtx_op_err () at /usr/src/lib/libthr/arch/amd64/amd64/_umtx_op_err.S:37 (gdb) bt #0 0x00000008009a2e8c in _umtx_op_err () at /usr/src/lib/libthr/arch/amd64/amd64/_umtx_op_err.S:37 #1 0x00000008009a1331 in cond_wait_common (cond=Variable "cond" is not available. ) at /usr/src/lib/libthr/thread/thr_cond.c:204 #2 0x00000000004c0573 in PyThread_acquire_lock (lock=0x70a760, waitflag=1) at thread_pthread.h:452 #3 0x00000000004c38b0 in lock_PyThread_acquire_lock (self=0x83f258, args=Variable "args" is not available. ) at ./Modules/threadmodule.c:46 #4 0x00000000004939e3 in PyEval_EvalFrameEx (f=0xa53920, throwflag=Variable "throwflag" is not available. ) at Python/ceval.c:3679 #5 0x00000000004939a8 in PyEval_EvalFrameEx (f=0xb57420, throwflag=Variable "throwflag" is not available. ) at Python/ceval.c:3765 #6 0x0000000000494ac1 in PyEval_EvalCodeEx (co=0x9aab70, globals=Variable "globals" is not available. ) at Python/ceval.c:2942 #7 0x00000000004eb6bc in function_call (func=0x9d8938, arg=0x9d9990, kw=0xa24da0) at Objects/funcobject.c:524 #8 0x0000000000417add in PyObject_Call (func=0x9d8938, arg=0x9d9990, kw=0xa24da0) at Objects/abstract.c:2487 #9 0x000000000048f730 in PyEval_EvalFrameEx (f=0xb48e20, throwflag=Variable "throwflag" is not available. ) at Python/ceval.c:3978 #10 0x00000000004939a8 in PyEval_EvalFrameEx (f=0xb57720, throwflag=Variable "throwflag" is not available. ) at Python/ceval.c:3765 #11 0x00000000004939a8 in PyEval_EvalFrameEx (f=0xb49020, throwflag=Variable "throwflag" is not available. ) at Python/ceval.c:3765 #12 0x0000000000494ac1 in PyEval_EvalCodeEx (co=0x947cd8, globals=Variable "globals" is not available. ) at Python/ceval.c:2942 #13 0x00000000004eb5bd in function_call (func=0x9a5320, arg=0x9d9950, kw=0x0) at Objects/funcobject.c:524 #14 0x0000000000417add in PyObject_Call (func=0x9a5320, arg=0x9d9950, kw=0x0) at Objects/abstract.c:2487 #15 0x000000000041f12f in instancemethod_call (func=0x9a5320, arg=0x9d9950, kw=0x0) at Objects/classobject.c:2558 #16 0x0000000000417add in PyObject_Call (func=0x93b410, arg=0x718050, kw=0x0) at Objects/abstract.c:2487 #17 0x000000000048d4c6 in PyEval_CallObjectWithKeywords (func=0x93b410, arg=0x718050, kw=0x0) at Python/ceval.c:3548 #18 0x00000000004c3cbd in t_bootstrap (boot_raw=0x70ae60) at ./Modules/threadmodule.c:425 #19 0x000000080099ac11 in thread_start (curthread=0xa61840) at /usr/src/lib/libthr/thread/thr_create.c:288 #20 0x0000000000000000 in ?? () Error accessing memory address 0x7fffff5fc000: Bad address. (gdb) thr 2 [Switching to thread 2 (Thread a61bc0 (LWP 100176))]#0 0x00000008009a2e8c in _umtx_op_err () at /usr/src/lib/libthr/arch/amd64/amd64/_umtx_op_err.S:37 37 RSYSCALL_ERR(_umtx_op) (gdb) bt #0 0x00000008009a2e8c in _umtx_op_err () at /usr/src/lib/libthr/arch/amd64/amd64/_umtx_op_err.S:37 #1 0x00000008009a1331 in cond_wait_common (cond=Variable "cond" is not available. ) at /usr/src/lib/libthr/thread/thr_cond.c:204 #2 0x00000000004c0573 in PyThread_acquire_lock (lock=0x70a600, waitflag=1) at thread_pthread.h:452 #3 0x00000000004c38b0 in lock_PyThread_acquire_lock (self=0x83f168, args=Variable "args" is not available. ) at ./Modules/threadmodule.c:46 #4 0x00000000004939e3 in PyEval_EvalFrameEx (f=0xb56520, throwflag=Variable "throwflag" is not available. ) at Python/ceval.c:3679 #5 0x0000000000494ac1 in PyEval_EvalCodeEx (co=0x8d5dc8, globals=Variable "globals" is not available. ) at Python/ceval.c:2942 #6 0x0000000000492c88 in PyEval_EvalFrameEx (f=0xb2de20, throwflag=Variable "throwflag" is not available. ) at Python/ceval.c:3775 #7 0x0000000000494ac1 in PyEval_EvalCodeEx (co=0x8d5288, globals=Variable "globals" is not available. ) at Python/ceval.c:2942 #8 0x00000000004eb5bd in function_call (func=0x9a8b18, arg=0xc524d0, kw=0x0) at Objects/funcobject.c:524 #9 0x0000000000417add in PyObject_Call (func=0x9a8b18, arg=0xc524d0, kw=0x0) at Objects/abstract.c:2487 #10 0x000000000041f12f in instancemethod_call (func=0x9a8b18, arg=0xc524d0, kw=0x0) at Objects/classobject.c:2558 #11 0x0000000000417add in PyObject_Call (func=0x93b370, arg=0x10331d0, kw=0x0) at Objects/abstract.c:2487 #12 0x0000000000464158 in slot_tp_init (self=Variable "self" is not available. ) at Objects/typeobject.c:5532 #13 0x0000000000461561 in type_call (type=0x7d9420, args=0x10331d0, kwds=0x0) at Objects/typeobject.c:700 #14 0x0000000000417add in PyObject_Call (func=0x7d9420, arg=0x10331d0, kw=0x0) at Objects/abstract.c:2487 #15 0x00000000004906f9 in PyEval_EvalFrameEx (f=0xb3a120, throwflag=Variable "throwflag" is not available. ) at Python/ceval.c:3890 #16 0x0000000000494ac1 in PyEval_EvalCodeEx (co=0x8d5eb8, globals=Variable "globals" is not available. ) at Python/ceval.c:2942 #17 0x0000000000492c88 in PyEval_EvalFrameEx (f=0xa47a20, throwflag=Variable "throwflag" is not available. ) at Python/ceval.c:3775 #18 0x0000000000494ac1 in PyEval_EvalCodeEx (co=0x8d5cd8, globals=Variable "globals" is not available. ) at Python/ceval.c:2942 #19 0x0000000000492c88 in PyEval_EvalFrameEx (f=0xb3a420, throwflag=Variable "throwflag" is not available. ) at Python/ceval.c:3775 #20 0x0000000000494ac1 in PyEval_EvalCodeEx (co=0x9aabe8, globals=Variable "globals" is not available. ) at Python/ceval.c:2942 #21 0x0000000000492c88 in PyEval_EvalFrameEx (f=0xb3a720, throwflag=Variable "throwflag" is not available. ) at Python/ceval.c:3775 #22 0x0000000000494ac1 in PyEval_EvalCodeEx (co=0x9aab70, globals=Variable "globals" is not available. ) at Python/ceval.c:2942 #23 0x00000000004eb6bc in function_call (func=0x9d8938, arg=0x9d98d0, kw=0xa5a660) at Objects/funcobject.c:524 #24 0x0000000000417add in PyObject_Call (func=0x9d8938, arg=0x9d98d0, kw=0xa5a660) at Objects/abstract.c:2487 #25 0x000000000048f730 in PyEval_EvalFrameEx (f=0xb2e020, throwflag=Variable "throwflag" is not available. ) at Python/ceval.c:3978 ---Type <return> to continue, or q <return> to quit--- #26 0x00000000004939a8 in PyEval_EvalFrameEx (f=0xb3aa20, throwflag=Variable "throwflag" is not available. ) at Python/ceval.c:3765 #27 0x00000000004939a8 in PyEval_EvalFrameEx (f=0xb2e220, throwflag=Variable "throwflag" is not available. ) at Python/ceval.c:3765 #28 0x0000000000494ac1 in PyEval_EvalCodeEx (co=0x947cd8, globals=Variable "globals" is not available. ) at Python/ceval.c:2942 #29 0x00000000004eb5bd in function_call (func=0x9a5320, arg=0x9d9890, kw=0x0) at Objects/funcobject.c:524 #30 0x0000000000417add in PyObject_Call (func=0x9a5320, arg=0x9d9890, kw=0x0) at Objects/abstract.c:2487 #31 0x000000000041f12f in instancemethod_call (func=0x9a5320, arg=0x9d9890, kw=0x0) at Objects/classobject.c:2558 #32 0x0000000000417add in PyObject_Call (func=0x93b320, arg=0x718050, kw=0x0) at Objects/abstract.c:2487 #33 0x000000000048d4c6 in PyEval_CallObjectWithKeywords (func=0x93b320, arg=0x718050, kw=0x0) at Python/ceval.c:3548 #34 0x00000000004c3cbd in t_bootstrap (boot_raw=0x70aee0) at ./Modules/threadmodule.c:425 #35 0x000000080099ac11 in thread_start (curthread=0xa61bc0) at /usr/src/lib/libthr/thread/thr_create.c:288 #36 0x0000000000000000 in ?? () Error accessing memory address 0x7fffff7fd000: Bad address. Apologies about the long backtraces [and thus the long message]. Thanks David
This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:39:35 UTC