On Fri, Jun 29, 2007 at 11:51:40AM +0100, David Malone wrote: > On Wed, Jun 27, 2007 at 06:43:11PM -0700, Steve Kargl wrote: > > Any advice on how to isolate or avoid? > > > > Jun 27 18:31:19 node11 kernel: TCP: [192.168.0.11]:59661 to > > [192.168.0.11]:63266 tcpflags 0x10<ACK>; syncache_expand: Segment failed > > SYNCOOKIE authentication, segment rejected (probably spoofed) > > It looks like you tried to open a TCP connection to yourself, but > the connection failed. You could try leaving a tcpdump running: > > tcpdump -i whatever_interface -w /tmp/synfinrstdata -s 1500 'tcp[tcpflags] & (tcp-syn|tcp-fin|tcp-rst) != 0' > > while your MPI app runs and then we can have a look at the packets > that caused the problem. The above should collect all TCP SYN, FIN > and RST packets, which would probably be enough to diagnose the > problem. > Another tidbit, once the MPI app started to trash, I ran truss on rank=0 process. I have a very file containing a sigprocmask(SIG_UNBLOCK,SIGCHLD,0x0) = 0 (0x0) poll({4/POLLIN 5/POLLIN 6/POLLIN 7/POLLIN 9/POLLIN 10/POLLIN 11/POLLIN 13/POLLIN 8/POLLIN 12/POLLIN 14/POLLIN 15/POLLIN 16/POLLIN 17/POLLIN 18/POLLIN 19/POLLIN 20/POLLIN 21/POLLIN 22/POLLIN 23/POLLIN 24/POLLIN 25/POLLIN 26/POLLIN 27/POLLIN 28/POLLIN 29/POLLIN 30/POLLIN},27,0) = 0 (0x0) sigprocmask(SIG_BLOCK,SIGCHLD,0x0) = 0 (0x0) sigaction(SIGCHLD,{ 0x3c0d2c850 SA_RESTART ss_t },0x0) = 0 (0x0) gettimeofday({1183138884.532826},0x0) = 0 (0x0) sched_yield(0x3c1d44180,0x3c0b39ec0,0x0,0x3c0b39ec0,0x3c1d44280) = 0 (0x0) sigprocmask(SIG_BLOCK,SIGCHLD,0x0) = 0 (0x0) sigaction(SIGCHLD,{ 0x3c0d2c850 SA_RESTART ss_t },0x0) = 0 (0x0) gettimeofday({1183138884.535137},0x0) = 0 (0x0) sigprocmask(SIG_UNBLOCK,SIGCHLD,0x0) = 0 (0x0) poll({4/POLLIN 5/POLLIN 6/POLLIN 7/POLLIN 9/POLLIN 10/POLLIN 11/POLLIN 13/POLLIN 8/POLLIN 12/POLLIN 14/POLLIN 15/POLLIN 16/POLLIN 17/POLLIN 18/POLLIN 19/POLLIN 20/POLLIN 21/POLLIN 22/POLLIN 23/POLLIN 24/POLLIN 25/POLLIN 26/POLLIN 27/POLLIN 28/POLLIN 29/POLLIN 30/POLLIN},27,0) = 0 (0x0) sigprocmask(SIG_BLOCK,SIGCHLD,0x0) = 0 (0x0) sigaction(SIGCHLD,{ 0x3c0d2c850 SA_RESTART ss_t },0x0) = 0 (0x0) gettimeofday({1183138884.538484},0x0) = 0 (0x0) ad nausem I'm using the 4BSD scheduler. -- SteveReceived on Fri Jun 29 2007 - 15:49:11 UTC
This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:39:13 UTC