packets duplicated *massively* on transmit.

Hi

I have two FreeBSD routers:

FreeBSD firewall1 7.0-CURRENT FreeBSD 7.0-CURRENT #7: Wed May 17 14:27:51 SAST 2006     ianf:/usr/obj/usr/src/sys/FIREWALL  i386

FreeBSD firewall2 7.0-CURRENT FreeBSD 7.0-CURRENT #8: Fri Sep  1 08:32:04 SAST 2006     ianf:/usr/obj/usr/src/sys/FIREWALL  i386

In two reasonably busy datacenters.  We're seeing packet loss that
we traced to a packet ariving on the world-facing interface being
retransmitted approximately every 10 microseconds or so for 1 to 5
seconds out of the interface the client is on.

Example trace:
Incoming packet on re0
1166597152.957627 00:02:85:07:32:40 > 00:30:4f:40:d9:cf, ethertype IPv4 (0x0800)
, length 62: 196.40.89.191.4655 > 196.40.102.12.445: S 1714709786:1714709786(0) 
win 8760 <mss 1460,nop,nop,sackOK>

Outbound packet(s) on vlan17 - parent re1
1166597153.000003 00:30:4f:40:d9:ee > 00:02:b3:d8:e7:4d, ethertype IPv4 (0x0800)
, length 62: 196.40.89.191.4655 > 196.40.102.12.445: S 1714709786:1714709786(0) 
win 8760 <mss 1460,nop,nop,sackOK>
1166597153.000013 00:30:4f:40:d9:ee > 00:02:b3:d8:e7:4d, ethertype IPv4 (0x0800)
, length 62: 196.40.89.191.4655 > 196.40.102.12.445: S 1714709786:1714709786(0) 
win 8760 <mss 1460,nop,nop,sackOK>
1166597153.000022 00:30:4f:40:d9:ee > 00:02:b3:d8:e7:4d, ethertype IPv4 (0x0800)
, length 62: 196.40.89.191.4655 > 196.40.102.12.445: S 1714709786:1714709786(0) 
win 8760 <mss 1460,nop,nop,sackOK>

We're seeing this on re(4) and rl(4) interfaces on firewall1 (uname
above) and on em(4) interfaces on firewall2.  It just transmits
faster on the em(4) interfaces.

Until recently, all instances I've seen so far had been SYN packets,
but I've just seen the same deal with an icmp echo request.  Sadly,
I don't have a copy of the original packet.

1166608024.000164 00:04:23:d4:7f:b3 > 00:01:29:19:06:c2, ethertype IPv4 (0x0800)
, length 98: (tos 0x0, ttl  28, id 16462, offset 0, flags [none], proto: ICMP (1
), length: 84) 196.22.132.223 > 196.22.138.62: ICMP echo request, id 34631, seq 
0, length 64
1166608024.000166 00:04:23:d4:7f:b3 > 00:01:29:19:06:c2, ethertype IPv4 (0x0800)
, length 98: (tos 0x0, ttl  28, id 16462, offset 0, flags [none], proto: ICMP (1
), length: 84) 196.22.132.223 > 196.22.138.62: ICMP echo request, id 34631, seq 
0, length 64
1166608024.000167 00:04:23:d4:7f:b3 > 00:01:29:19:06:c2, ethertype IPv4 (0x0800)
, length 98: (tos 0x0, ttl  28, id 16462, offset 0, flags [none], proto: ICMP (1
), length: 84) 196.22.132.223 > 196.22.138.62: ICMP echo request, id 34631, seq 
0, length 64
1166608024.000169 00:04:23:d4:7f:b3 > 00:01:29:19:06:c2, ethertype IPv4 (0x0800)
, length 98: (tos 0x0, ttl  28, id 16462, offset 0, flags [none], proto: ICMP (1
), length: 84) 196.22.132.223 > 196.22.138.62: ICMP echo request, id 34631, seq 
0, length 64

Any ideas?

Ian

--
Ian Freislich