Re: Some interesting plots

Randall Stewart wrote:
> All:
> 
> As you all may know I have been working on getting SCTP
> into Current..
> 
> I have been, of late, trying to tweak things to get
> the BEST performance out of the implementation...

Randall,

get the code correct first, check.  Then second get it into the
tree.  Third make it fast.  Once it is in the tree we can work
out kinks much easier.  Testing gets easier too.  I've got a lot
of experience with socket buffer performance and interactions
now and can help as soon as it is in the tree.

It's just not necessary to have it absolutely perfect when it
goes into the tree.  FreeBSD-current has another 8 month to shake
things out.

-- 
Andre

> I have moved my testing off between two 7.0 machines.. same
> code base on each updated Sep 25.
> 
> One is a 2.8Gig Dell SC1600 Xeon.. (hyper threaded CPU).
> The other is a P4D (2.8Gig .. slightly faster, true dual
>                      processor machine).
> 
> They are connected by two intel EM server cards like so:
> 
> 
> +----+                                 +----+
>   1  | em1 <---------------------> em0 | 2
>      | em0 <-----locallan--------> msk0|
>      | dc0 <-Direct Inet               |
> +----+                                 +-----+
> 
> 
> em1 has 10.1.2.12                   em0 10.1.2.21
> em0 has 10.1.1.12                  msk0 10.1.1.21
> 
> 
> Now I setup the association with a server on 1, the
> conversation goes something like this:
> 
>                                             gettimeofday()
>      <---------------Send me N bytes in X byte records----
>     ------>send---->
>     ------>send---->                          when N bytes are recieved
>                                                   gettimeofday()/close()
>     ...
>     close()
>                                             calculate and announce b/w
> 
> Now the SCTP implemenation has some interesting logging things that
> allow me to turn on various compile options that log into a 80,000 entry
> memory buffer that you can pull with specific commands.
> 
> I have rigged the sender to log each sack in and tell
> on the sender side (1):
> 
> a) How much data is queued to the peer
> b) Total flight size
> c) rwnd advertised
> d) the calulated rwnd.
> 
> On the receiver side (2) on both exit and entry
> to the receiving function in the kernel:
> On entry:
> a) read request size (uio_resid)
> b) amount in queue on the sb to read
> c) if blocking is allowed
> d) How much has to be read total to get a window
>    update sack out
> On exit:
> a) how much has been read since the last sack sent
> b) how much is left un-read
> c) the rwnd we are advertising
> d) how much is still on the sb.
> 
> ------------------------------------------------------------
> 
> The logging is nicely light weight (as long as you don't roll the
> log). This is because I use some atomic functions to move forward
> so I have picked a size that won't roll it...
> 
> I have then played a bit with GNU plot to generate some
> graphs of all this... fun fun ;-D
> 
> Now I have placed the plots at:
> 
> http://www.sctp.org/plots.tar.bz2
> 
> (they are about 2Meg.. and I have a 512k up link .. so patience is
>  a must).
> 
> So what will you see..
> 
> plots that say *plotall.ps give you the entire transfer.
> 
> ones that say
> 
> plot*.ps show everything:
> 
> a) When things are read in red y axis is size unread
> b) what the sender has in flight in green
> c) what the a-rwnd is in blue
> d) send-q size in pink.
> 
> There is always a plot for each 100ms so you
> can "zoom" in on things...
> Especially 1.4-1.5 seconds look interesting..
> 
> Now plots that have
> cntplot*
> 
> Show
> 
> a) flight
> b) arwnd
> c) outq
> and
> d) number of sacks every 400us (multiplied by 1000 so
>    if you had 10 sacks in a 400us period you would show
>    it at 40000).
> 
> This too is interesting.
> 
> Finally plots that have lcntplot*
> show only the flight size vs
> the number of sack's every 500us (multiplied
> by 1000)...
> 
> The interesting thing that I have going on is:
> 
> 1) The sender is running dry.. you can see this in the graphs
>    when the flight crashes down to almost nothing at say
>    1.5 seconds or so... also seen whenever the blue line
>    moves down to the green flight..
> 
> 2) Any time the send queue (blue line) is at the flight (green)
>    this is bad since it means we have the danger of running dry.
> 
> 3) The reader is completely keeping up... since it
>    reads constantly and never has much of a sb_cc (the
>    red line stays on the bottom of the page).
> 
> Now whats strange about this is that I am only getting 760Megbits per
> second (or so).. and in that 760Mb I see processor <1> about running
> 60-70% idle.. and processor <2> running 70-80% idle.
> 
> Now I am most puzzeled on how to make this run faster...
> 
> a) what causes the spikes (ideas would be welcome)
> b) why can't I get more CPU.. I have seperate locks that
>    the receiver and the sender use to make it so there is
>    minimal contention between the network side of things and
>    the queuing and or reading... In fact I just added the sender
>    seperate lock in the last day or so.. This is because
>    I was seeing not 2 or so but 8 dips before I did that in
>    the "queue"
> 
> One other note, I see TCP is only getting 250Meg or so on the
> same test (It can run either).. now it used to get close to
> the full pipe (Gigbit).. so is there some issue with the new code
> that was recently submitted?
> 
> Have a look.. its interesting.. I am rather clueless
> though on how to grab that CPU that top tells me is
> there (if its accurate :-D) suggestions on where to look
> or tuning I could do would be welcome.
> 
> Thanks
> 
> 
> R