Re: how to use the ktls

John Baldwin wrote:
>On 1/26/20 8:08 PM, Rick Macklem wrote:
>> John Baldwin wrote:
>> [stuff snipped]
>>> Hmmm, this might be a fair bit of work indeed.
>>>
>>> Right now KTLS only works for transmit (though I have some WIP for receive).
>>>
>>> KTLS does assumes that the initial handshake and key negotiation is handled by
>>> OpenSSL.  OpenSSL uses custom setockopt() calls to tell the kernel which
>>> session keys to use.
>>>
>>> I think what you would want to do is use something like OpenSSL_connect() in
>>> userspace, and then check to see if KTLS "worked".  If it did, you can tell
>>> the kernel it can write to the socket directly, otherwise you will have to
>>> bounce data back out to userspace to run it through SSL_write() and have
>>> userspace do SSL_read() and then feed data into the kernel.
>>>
>>> The pseudo-code might look something like:
>>>
>>> SSL *s;
>>>
>>> s = SSL_new(...);
>>>
>>> /* fd is the existing TCP socket */
>>> SSL_set_fd(s, fd);
>>> OpenSSL_connect(s);
>>> if (BIO_get_ktls_send(SSL_get_wbio(s)) {
>>>    /* Can use KTLS for transmit. */
>>> }
>>> if (BIO_get_ktls_recv(SSL_get_rbio(s)) {
>>>    /* Can use KTLS for receive. */
>>> }
>>
>> So, I've been making some progress. The first stab at the daemons that do the
>> handshake are now on svn in base/projects/nfs-over-tls/usr.sbin/rpctlscd and
>> rpctlssd.
>>
>> A couple of questions...
>> 1 - I haven't found BIO_get_ktls_send() or BIO_get_ktls_recv(). Are they in some
>>        different library?
>
>They only existing currently in OpenSSL master (which will be OpenSSL 3.0.0 when it
>is released).  I have some not-yet-tested WIP changes to backport those changes into
>the base OpenSSL, but it will also add overhead to future OpenSSL imports perhaps,
>so it is something I need to work with secteam_at_ on to decide if it's viable once I
>have a tested PoC.
>
>I will try to at least provide a patch to the security/openssl port to add a KTLS
>option "soon" that you could use for testing.
John, I wouldn't worry much about this.
The calls are currently #ifdef notnow in the daemon and I'm fine with that.
SSL_connect() has returned 1, so the daemon knows that the handshake is complete and
the kernel code that did the upcall to the daemon can check for KERN_TLS support.

>> 2 - After a successful SSL_connect(), the receive queue for the socket has 478bytes
>>       of stuff in it. SSL_read() seems to know how to skip over it, but I haven't
>>       figured out a good way to do this. (I currently just do a recv(..478,0) on the
>>       socket.)
>>       Any idea what to do with this? (Or will the receive side of the ktls figure out
>>       how to skip over it?)
>
>I don't know yet. :-/  With the TOE-based TLS I had been testing with, this doesn't
>happen because the NIC blocks the data until it gets the key and then it's always
>available via KTLS.  With software-based KTLS for RX (which I'm going to start
>working on soon), this won't be the case and you will potentially have some data
>already ready by OpenSSL that needs to be drained from OpenSSL before you can
>depend on KTLS.  It's probably only the first few messsages, but I will need to figure
>out a way that you can tell how much pending data in userland you need to read via
>SSL_read() and then pass back into the kernel before relying on KTLS (it would just
>be a single chunk of data after SSL_connect you would have to do this for).
Well, SSL_read() doesn't return these bytes. I think it just throws them away.

I have a simple test client/server where the client sends "HELLO THERE" to the
server and the server replies "GOODBYE" after the SSL_connect()/SSL_accept()
has been done.
--> If the "HELLO THERE"/"GOODBYE" is done with SSL_write()/SSL_read() it works.
however
--> If the above is done with send()/recv(), the server gets the "HELLO THERE", but
      the client gets 485bytes of data, where the last 7 are "GOODBYE".
      --> If I do a recv( ..475..) in the client right after SSL_connect() it works ok.

I do this for testing, since it can then do the NFS mount (unencrypted).

Looking inside SSL_read() I found:
*
1742	     * If we are a client and haven't received the ServerHello etc then we
1743	     * better do that
1744	     */
1745	    ossl_statem_check_finish_init(s, 0);

but all ossl_statem_check_finish_init(s, 0); seems to do is set a variable "in_init = 1".
Then it calls the method->read() function, which I'd guess is in the crypto code
and it seems to get rid of these bytes. (I hate trying to figure out what calls what
for object oriented code;-).

Btw. SSL_is_finished_init() returns 1 right after the SSL_connect(), so it seems to
think the hadnshake is done, although it hasn't read these 478 bytes from the server.

Anyhow, I'd guess the TOE engine knows how to do get rid of this stuff like SSL_read()
does?

>> I'm currently testing with a kernel that doesn't have options KERN_TLS and
>> (so long as I get rid of the 478 bytes), it then just does unencrypted RPCs.
>>
>> So, I guess the big question is.... can I get access to your WIP code for KTLS
>> receive? (I have no idea if I can make progress on it, but I can't do a lot more
>> before I have that.)
>
>The WIP only works right now if you have a Chelsio T6 NIC as it uses the T6's TCP
>offload engine to do TLS.  If you don't have that gear, ping me off-list.  It
>would also let you not worry about the SSL_read case for now for initial testing.
I may contact you in April about this.
Since you've noted above that you haven't started the software version, I might try
coming up with something and will let you know if I do.

Thanks, rick