Re: IPSEC disables TSO

On Thu, 16 Aug 2007, Andrew Gallatin wrote:

> > I thought about this after sending and realized that the appropriate 
> > response is that IPSEC needs to be careful to disable TSO when its in use 
> > for a connection. What you're seeing was most likely done as an expedient.
>
> Yes, exactly, there needs to be a smarter test that can distingiush if IPSEC 
> is actually in use on a connection or not; I should have been more clear 
> about this.  The problem is that I have zero knowledge about IPSEC, so I 
> have no idea how to do this.
>
> I'm worried that people will compile IPSEC into the kernel to run an 
> encrypted tunnel (or the TCP MD5 signature stuff for BGP), and then be 
> rather surprised that their their "normal" TCP performance stinks.

There are a few things going on here, and a few different approaches that 
could be taken.  I'm pretty sure we have problems with the handling of various 
TCP TSO edge cases.

Right now we handle one "disabling" edge case, and the way it's handled may be 
instructive in thinking about the problem: if a route changes, or TSO is 
disabled on the interface, EMSGSIZE will be returned from the IP output 
function, causing TCP to reevaluate the segmentation strategy it's using, and 
turn off TSO as required.  This case may not properly handle IPSEC as it runs 
today, because it's reevaluation of whether to use TSO and what MTU size to 
use likely doesn't take IPSEC into account.  However, you could imagine that 
it might in the future, in which case the right way to handle this case might 
be for IPSEC to simply encrypt the datagram and let the EMSGSIZE from its send 
propagate back up to IP and then TCP.

Another way to deal with this whole scenario is to treat TSO the same way we 
treat offloaded checksums: provide a just-in-time segmentation routine to run 
at the last moment when we discover that TSO is not the right strategy.  This 
isn't, probably, generally the right solution (i.e., don't always use TSO at 
the TCP layer), as we'd like TCP to make the segmentation decisions with more 
information in the general case (so you don't end up with awkwardly sized 
segments, etc).  However, there may be other places where it does make sense.

We also currently have no way to enable/re-enable TSO in the event that we 
start routing to an interface that has TSO enabled, or where TSO is turned on 
administratively, so once a connection has started, it can only ever downgrade 
from TSO operation to normal operation, never reverse the change.  For some 
protocols, this is undesirable -- for example, for long-running NFS TCP 
connections, it would be nice if TSO would turn itself back on again. 
However, we should get basic TSO working properly before turning to issues 
like this :-).

Robert N M Watson
Computer Laboratory
University of Cambridge