Re: mpi + shmem issues

additional information:

when recompiling mpich for debugging symbols, configure fails on:
checking that usable shared memory locks were found... no

so, does this mean that mpich somehow exhausted all shmem locks?
after running the program only 10 times, i see this as infeasible,
considering
	a) mpich (presumably in MPI_Init()) would only want 1 or
	   2 locks on init
and
	b) any shared memory locks mpich grabs should be freed
	   upon process completion (whether clean or not) by the
	   operating system, no?

well, configure with --with-device=ch_shmem:-usesysv succeeds
(checking that usable shared memory locks were found... yes), so
I will try this out.

But for reference, can anyone make a guess as to why/how a shared
memory application can exhaust locks like this?

thank you.

-Anthony.


On Sun, May 25, 2003 at 02:49:30AM -0400, Anthony Schneider wrote:
> Hello,
> My machine is a dual athlon:
> FreeBSD pickle. 5.1-BETA FreeBSD 5.1-BETA #6: Sun May 25 02:16:15 EDT 2003
> anthony_at_pickle.:/usr/src/sys/i386/compile/PICKLE  i386
> 
> I started having this issue, which may or may not exist on uniprocessor
> systems or 4.x systems.  I built mpi with ch_shmem device for shared memory
> programs (instead of the more common rsh/ssh), and something strange
> happens.  For even the most basic little program, the program will launch
> fine (usually) the first time i run it after the system boots, but after a
> few executions, execution starts failing consistently until after i reboot.
> 
> as an example, here is a small acknowledgment program:
> 
> #include <mpi.h>
> #include <stdio.h>
> 
> int main (int argc, char *argv[]) {
>         int mpiRank, mpiSize;
> 
>         MPI_Init (&argc, &argv);
>         MPI_Comm_rank (MPI_COMM_WORLD, &mpiRank);
> 
>         printf ("#%d here\n", mpiRank);
> 
>         return 0;
> 
> }
> 
> and here is the history of executing it:
> 
> pickle:anthony:/home/anthony/src/mpi:6% mpirun -np 2 ./foo
> #0 here
> #1 here
> Child process exited unexpectedly 0
> Abort trap (core dumped)
> pickle:anthony:/home/anthony/src/mpi:7% mpirun -np 2 ./foo
> #0 here
> pickle:anthony:/home/anthony/src/mpi:8% #1 here
> 
> pickle:anthony:/home/anthony/src/mpi:8% mpirun -np 2 ./foo
> #0 here
> #1 here
> pickle:anthony:/home/anthony/src/mpi:9% mpirun -np 2 ./foo
> #0 here
> #1 here
> pickle:anthony:/home/anthony/src/mpi:10% mpirun -np 2 ./foo
> #1 here
> #0 here
> Child process exited unexpectedly 0
> Abort trap (core dumped)
> pickle:anthony:/home/anthony/src/mpi:11% mpirun -np 2 ./foo
> #0 here
> #1 here
> Child process exited unexpectedly 0
> Abort trap (core dumped)
> pickle:anthony:/home/anthony/src/mpi:12% mpirun -np 2 ./foo
> #0 here
> #1 here
> pickle:anthony:/home/anthony/src/mpi:13% mpirun -np 2 ./foo
> #1 here
> #0 here
> Child process exited unexpectedly 0
> Abort trap (core dumped)
> pickle:anthony:/home/anthony/src/mpi:14% mpirun -np 2 ./foo
> #0 here
> #1 here
> pickle:anthony:/home/anthony/src/mpi:15% mpirun -np 2 ./foo
> #0 here
> #1 here
> pickle:anthony:/home/anthony/src/mpi:16% mpirun -np 2 ./foo
> semget failed for setnum =  0
> Abort trap (core dumped)
> pickle:anthony:/home/anthony/src/mpi:17% mpirun -np 2 ./foo
> semget failed for setnum =  0
> Abort trap (core dumped)
> pickle:anthony:/home/anthony/src/mpi:18% mpirun -np 2 ./foo
> semget failed for setnum =  0
> Abort trap (core dumped)
> 
> ... (continues until i reboot)
> 
> the first run that aborts is strange, but since it is not something
> i've witnessed previously, i'd like to forget that and focus on
> the repeated semget failures.  i would normally be looking into
> the mpi implementation (mpich 1.2.5), but since after semget fails
> once it never seems to succeed again with other mpi programs, i
> think this to be a freebsd problem.
> 
> i'm runing a (barely) custom kernel, with nothing added to it.
> i just cvsup'd and rebuilt less than an hour ago, and the problem
> has persisted from beta #5 through beta #6.
> 
> any suggestions?
> 
> thank you for your help.
> 
> -Anthony.