Re: Strange things on GBit / 1000->100 / net.inet.tcp.inflight.*

On Sat, Sep 18, 2004 at 03:15:07AM +0200, Andre Oppermann wrote:
> > I'm not sure this is a "physical" problem.
> 
> The high number of out-of-order packets on your 4.10 machine is very
> suspicious.

You're talking about
http://rhb.uugrn.org/FreeBSD/bugs/5.x/1000-to-100/check_3_target.txt ?

Maybe the switch is buggy for just this testcase? Store&forward-Engine?

> > Workaround:
> > * Buy 1000MBit-Cards for the both "Router"-boxes
> Why that?  Do you have to do TCP transfer to/from these machines?

The two machies are routers, but ... 
 
> Yet another test you could try:
>  5.3 GE ---> FE 4.10 "Router" FE ---> 4.10 FE

This was my initial testcase, I tried to transfer data from the GE-net
via the "internal" router to another internal FE-4.x-Box --> 700kb/sec IIRC
(Intranet)
That failed. My first suspicion was the internal router (10.101.240.1,
4.10, FE) is not able to route a higher bandwidth for any reason. I tried 
something via the router. To mask some other unknown bottlenecks I tried 
direkt from the router (as target, not as router) ... now here we are.

>From my point of view  ther's no significant difference between "router
just routes" or "router is target". Maybe I'm wrong here.

Just tested the case via the 2nd router to another net.

wget -O - --proxy=off ftp://10.101.240.54/pub/1GB  > /dev/null
Länge: 1,160,773,632 (unmaßgeblich)

 2% [=>   ] 28,918,448   582.63K/s    ETA 31:37^C

 1  145.2xx.xxx.1x6  0.243 ms  0.120 ms  0.133 ms
 2  10.101.240.54  0.357 ms  0.229 ms  0.343 ms

Same switch, different irons (target, router, server), another nets, ...

Maybe interesting: wget starts with about 1-2MBytes/sec then converging
down to  some 100kBytes/sec. 

Another comparison, differrent targets (4.10 FE, 5.3GE), different
servers (4.x,5.3 GE), see PS for a detailed list:

1) target: external router (10.101.240.254, 4.10, FE)
$ wget -O - --proxy=off ftp://10.101.240.52/pub/1GB  > /dev/null --> 9.42M/s
$ wget -O - --proxy=off ftp://10.101.240.1/pub/250M > /dev/null  --> 10.22M/s
$ wget -O - --proxy=off ftp://10.101.240.54/pub/1GB  > /dev/null --> 258.47K/s
$ wget -O - --proxy=off ftp://10.101.240.55/pub/1GB  > /dev/null --> 384.70K/s
$ wget -O - --proxy=off ftp://10.101.240.56/pub/1GB  > /dev/null --> 195.71K/s

2) target: neighbor-server (10.101.240.55, 5.3,GE)
$ wget -O - --proxy=off ftp://10.101.240.52/pub/1GB  > /dev/null --> 55.55M/s 
(uncached, the .52 cannot cache 1GByte file)
$ wget -O - --proxy=off ftp://10.101.240.54/pub/1GB  > /dev/null --> 34.12M/s (uncached from RAID)
$ wget -O - --proxy=off ftp://10.101.240.54/pub/1GB  > /dev/null --> 76.96M/s (cached)
$ wget -O - --proxy=off ftp://10.101.240.56/pub/1GB  > /dev/null --> 36.61M/s (uncached from RAID)
$ wget -O - --proxy=off ftp://10.101.240.56/pub/1GB  > /dev/null --> 77.40M/s (cached)
$ wget -O - --proxy=off ftp://10.101.240.254/pub/250M  > /dev/null --> 10.07M/s
$ wget -O - --proxy=off ftp://10.101.240.1/pub/250M  > /dev/null --> 10.37M/s

I guess the switch has some troubles with its store&forward engine when
transferring data from GE to FE and(!) the GE is a 5.3, if GE is 4.10
then it works. So the only significant difference is the OS (thats why 
I discuss it here and not with the guys of netgear this time).

I have a spare "router-box", running a  offline-hot-standby copy of our 
external packet filter (outside our external DMZ). 
I may install 5.3 on that box to have a 5.3 running on FE, even though 
I don't belive that will make a difference what kind of system is the 
target. Who knows.


Regards
Raphael Becker


PS: all players on GE:

IP   OS      IF       desc      
.1   4.10    fxp1,FE  internal router, PE350, 128MB RAM
.254 4.10    fxp1,FE  external router, PE350, 128MB RAM
.52  4.10    bge1,GE  normal host, PE2650, 1024MB RAM
.54  5.3     bge0,GE  dito, 2048MB RAM
.55  5.3     bge0,GE  dito, 2048MB RAM
.56  5.3     bge0,GE  dito, 2048MB RAM