Re: Getting started with ktls

On Wed, Mar 10, 2021 at 06:17:42PM -0700, Alan Somers wrote:
> On Wed, Mar 10, 2021 at 5:31 PM Benjamin Kaduk <kaduk_at_mit.edu> wrote:
> 
> > On Wed, Mar 10, 2021 at 05:18:24PM -0700, Alan Somers wrote:
> > > I'm trying to make ktls work with "zfs send/recv" to substantially reduce
> > > the CPU utilization of applications like zrepl.  But I have a few
> > questions:
> > >
> > > * ktls(4)'s "Transmit" section says "Once TLS transmit is enabled by a
> > > successful set of the TCP_TXTLS_ENABLE socket option", but the "Supported
> > > Libraries" section says "Applications using a supported library should
> > > generally work with ktls without any changes".  These sentences seem to
> > be
> > > contradictory.  I think it means that the TCP_TXTLS_ENABLE option is
> > > necessary, but OpenSSL sets it automatically?
> >
> > Yes, OpenSSL sets it automatically for the builtin socket and connection
> > BIO classes.  Applications using other BIO classes will need to do things
> > manually (or implement the appropriate _ctrl() parameters for their BIO
> > class).
> >
> > > * When using OpenSSL, the library will automatically call setsockopt(_,
> > > TCP_TXTLS_ENABLE).  But it swallows the error, if any.  How is an
> > > application to tell if ktls is enabled on a particular socket or OpenSSL
> > > session?
> >
> > IIRC the lack of answer for this is part of why upstream OpenSSL doesn't
> > have specific KTLS tests enabled in the automated test suite.
> >
> 
> getsockopt(_. TCP_TXTLS_ENABLE) returns ENOPROTOOPT.  Is there any reason
> why it's not implemented?  That might be the easiest way to check for the
> ktls status of an individual socket.

I think that's probably more of a question for jhb than me.  I don't know
of a reason why not, but I do know that there is some desire to keep the
functionality that openssl exposes roughly compatible between linux and
FreeBSD KTLS.  I don't know whether Linux has something similar.

> 
> >
> > > * From experiment, I can see that OpenSSL attempts to set
> > > TCP_TXTLS_ENABLE.  But it doesn't try to set TCP_RXTLS_ENABLE.  Why not?
> > > From reading ktls_start and ossl_statem_server_post_work, it looks like
> > > maybe a single socket cannot have ktls enabled for both sending and
> > > receiving at the same time.  Is that true?
> >
> > No.  They just get enabled separately, since change_cipher_state() is
> > called separately for read and write transitions.
> >
> 
> Apologies if I'm too ignorant, but what is a transition in SSL-speak?  This
> is my first attempt at any kind of SSL programming.  What I know from
> ktrace is that TCP_RXTLS_ENABLE never gets set.

Sorry!  I'm pretty conversant with this stuff (I'm the security area
director that is responsible for the IETF TLS working group) and don't
always target the right level.  Basically, for a decent encrypting protocol
you want different encrytion keys for the read and write direction
(whichever peer you are), and the TLS (1.3) handshake has a multi-stage key
hierarchy to try to encrypt as much of it as possible.  So, for example,
the client will need to update it's encryption key for reading once it
reads the ServerHello (and before reading the Encrypted Extensions)
message, even though the keys the client uses for writing don't change at
that time.  Internally, OpenSSL implements this "transition" of key
material with a change_cipher_state() abstraction, that takes a flags
argument (`which`).  The flags indicate which set of keys to update, and
which direction (read or write).  So, by my read of the code, what's
*supposed* to happen is that we call:

    if (BIO_set_ktls(bio, &crypto_info, which & SSL3_CC_WRITE))

And if SSL3_CC_WRITE is set, that translates to calling BIO_set_ktls() with
an `is_txt` value that evaluates to true; otherwise, `is_txt` is false,
which corresponds to the RX case that you're failing to see happen.

Just to get the boring stuff out of the way: what version of openssl are
you testing against, and did you verify that OPENSSL_NO_KTLS_RX is not
defined when ktls_start() is being compiled (so that the setsockopt(fd,
IPPROTO_TCP, TCP_RXTLS_ENABLE, .) is compiled in at all)?

Thanks,

Ben