Strange NFS behaviour

Hoi,

I'm not shure what to think of this...
Looking at building a NFS/Samba server to stick somewhere in a
corner in our house, I ran a few "performance" tests. If you would
like to call bonnie(1) such a beast. I used bonnie to get a first
feel for things, not as something I'd like to publish a paper with.

I just varied the size of the of the read and write block with UDP
and TCP. Slightly against what mount_nfs advises. But the results are
not very pleasing:

              -------Sequential Output-------- ---Sequential
Input-- --Random--
              -Per Char- --Block--- -Rewrite-- -Per
Char- --Block--- --Seeks---
Machine    MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU  /sec
%CPU
local     256   759 10.1   886  1.1   750  2.4  4280 51.6  5383  7.1  96.3
2.8
udp-1k    256   629  9.3   638  1.0   400  4.2  1533 19.1  1522  2.9  79.6
4.7
udp-2k    256   718 10.2   714  1.1   488  3.5  2144 27.0  2561  4.8  68.3
3.4
udp-4k    256   774 10.7   783  1.2   551  3.1  3299 41.6  4379  8.0  79.9
3.1
udp-8k    256   757 10.1   807  1.0   765  2.4  3703 44.9  5147  7.3  84.0
2.7
udp-16k   256  1199 15.7  1137  1.3  1062  3.1  4525 54.5  5609  6.2  83.9
2.8
udp-32k   256  1215 15.8  1155  1.3  1067  3.1  4215 50.8  5629  6.4  86.1
3.0
udp-64k   256  1180 15.4  1137  1.3  1060  3.1  3815 46.0  5629  6.3  86.3
3.2
tcp-64k   256  1211 16.5  1002  1.2   245  0.8  4523 55.2  5689  5.8  61.5
3.9
tcp-64k   256  1160 15.8  1013  1.3   243  0.8  3971 48.1  5418  5.4  58.1
3.8
tcp-32k   256  1187 16.1   959  1.2   243  0.8  4088 49.6  4707  4.6  52.9
3.6
tcp-32k   256  1190 16.1   992  1.2   242  0.8  4204 51.2  5310  5.2  52.9
3.6
tcp-16k   256   885 11.9   798  0.9   139  0.5  3955 48.0  4576  5.8  76.3
3.6
tcp-16k   256   921 12.4   817  1.0   138  0.5  4241 51.5  4765  6.1  56.6
2.7
tcp-8k    256   750 10.0   844  1.0    75  0.3  3317 41.0  4140  6.4  53.1
2.1
tcp-8k    256   684  9.2   774  0.9    76  0.3  2895 35.7  4108  6.4  76.3
3.0
tcp-4k    256   758 10.2   734  1.1    74  0.6  2658 33.5  3170  6.1  69.0
3.5
tcp-4k    256   715  9.6   698  1.0    74  0.6  2618 33.1  3194  6.2  67.6
3.3

local is when I run bonnie on the local file system/server, which runs
5.1-RELEASE-p11/GENERIC on a P3-233mhz with 96Mb RAM and a 4 disk 170Gb
Vinum-Raid5.
(Which gives a feel on why the local number as so low while writing)
Other lines give the transfers using either TCP or UDP with 1K^2n block
size.

Now what strikes me as very odd, is the huge difference between TCP and UDP
in
the rewrite column. And, yes I'm aware of the fact that for every test
config
there is a setup that is going to perform poorly. But the "only" difference
is
the type if "virtual-wire" between the machines.

But such a huge difference????

BTW the client box runs: 5.2-CURRENT Mon Dec 22, and is a P3-533, with 256
Mb,
and has the diagnostics stuff off.
Both systems have an Intel based 259 fxp0 card set to 100Mbit, full-duplex,
connected by a simple switch.

I've found very little of errors, or unusual things.
On the client in netstat -s:
tcp:
    5537543 segments updated rtt (of 2029581 attempts)
on the server:
tcp:
    4787548 segments updated rtt (of 1864984 attempts)
Which is something that I can not really place.
Very little is dropped, retransmitted or out of sequence.

So the 1$ question is:
    Why is this difference there, and should I(we) worry about it.

Note that I'm going to run this as well with a "real" workload:
    compiling a kernel
But this was just a way of getting a flavour of things.

--WjW