Re: FBSD 1GBit router?

--- Ingo Flaschberger <if_at_xip.at> wrote:

> 
> >> I have a 1.2Ghz Pentium-M appliance, with 4x
> 32bit, 33MHz pci intel e1000 
> >> cards.
> >> With maximum tuning I can "route" ~400mbps with
> big packets and ~80mbps 
> >> with 64byte packets.
> >> around 100kpps, whats not bad for a pci
> architecture.
> >> 
> >> To reach higher bandwiths, better busses are
> needed.
> >> pci-express cards are currently the best choice.
> >> one dedicated pci-express lane (1.25gbps) has
> more bandwith than a whole 
> >> 32bit, 33mhz pci-bus.
> >
> > Like you say routing 400 Mb/s is close to the max
> of the PCI bus, which
> > has a theoretical max of 33*4*8 ~ 1Gbps. Now
> routing is 500Mb/s in, 500Mb/s 
> > out. So you are within 80% of the bus-max, not
> counting memory-access and 
> > others.
> 
> yes.
> 
> > PCI express will give you a bus per PCI-E device
> into a central hub, thus 
> > upping the limit to the speed of the FrontSideBus
> in Intel architectures. 
> > Which at the moment is a lot higher than what a
> single PCI bus does.
> 
> Thats why my next router will be based at this box:
>
http://www.axiomtek.com/products/ViewProduct.asp?view=429
> 
> Hopefully there will be direct memory bus connected
> nic's in future.
> (HyperTransport connected nic's)
> 
> > What it does not explain is why you can only get
> 80Mb/s with 64byte packets, 
> > which would suggest other bottlenecks than just
> the bus.

To clarify this, you seem to leave out PCI-X, which is
an 8Mb/s bus, which is certainly able to route or
bridge a gigabit of traffic. PCIe 4x has an unencoded
data rate of 16Gb/s and PCIe 8x is 64Gb/s, which is
more than enough to do gigabit and 10gig. 

PCI can BURST to 1Gb/s and PCI-X can BURST to 8Gb/s,
but bursts are limited, so its not possible to achieve
full bandwidth. The more devices on the bus, the less
throughput you'll get due to contention.

A more limiting factor for routing is packets/second,
not the actual throughput. Its foolhardy to use a NIC
that doesn't have enough bandwidth, so "usually", the
limiting factor is the CPUs ability to process the
packets regardless of their size.

We bridged 1 million PPS on a FreeBSD 4.x machine with
a 2.8Ghz opteron and PCI-x intel cards. A 7.0 system
can do about 20% less (of course you don't lose the
keyboard on a 7.0 system as you do on the 4.x
system!). So I'd assume a 3Ghz xeon could likely do
close to 1 million pps on a 7.0 system. Routing will
be a bit slower.

Also be advised that implementation is an issue with
bus througput.  I've tested systems with both PCIe and
PCIx and the PCIx gave higher thoughput even though
the PCIe is theoretically faster. MB/chipset design is
a factor in the utilization capability of the bus.

Barney


      ____________________________________________________________________________________
Be a better friend, newshound, and 
know-it-all with Yahoo! Mobile.  Try it now.  http://mobile.yahoo.com/;_ylt=Ahu06i62sR8HDtDypao8Wcj9tAcJ