Re: Getting started with ktls

On Wed, Mar 10, 2021 at 8:15 PM Benjamin Kaduk <kaduk_at_mit.edu> wrote:

> On Wed, Mar 10, 2021 at 06:17:42PM -0700, Alan Somers wrote:
> > On Wed, Mar 10, 2021 at 5:31 PM Benjamin Kaduk <kaduk_at_mit.edu> wrote:
> >
> > > On Wed, Mar 10, 2021 at 05:18:24PM -0700, Alan Somers wrote:
> > > > I'm trying to make ktls work with "zfs send/recv" to substantially
> reduce
> > > > the CPU utilization of applications like zrepl.  But I have a few
> > > questions:
> > > >
> > > > * ktls(4)'s "Transmit" section says "Once TLS transmit is enabled by
> a
> > > > successful set of the TCP_TXTLS_ENABLE socket option", but the
> "Supported
> > > > Libraries" section says "Applications using a supported library
> should
> > > > generally work with ktls without any changes".  These sentences seem
> to
> > > be
> > > > contradictory.  I think it means that the TCP_TXTLS_ENABLE option is
> > > > necessary, but OpenSSL sets it automatically?
> > >
> > > Yes, OpenSSL sets it automatically for the builtin socket and
> connection
> > > BIO classes.  Applications using other BIO classes will need to do
> things
> > > manually (or implement the appropriate _ctrl() parameters for their BIO
> > > class).
> > >
> > > > * When using OpenSSL, the library will automatically call
> setsockopt(_,
> > > > TCP_TXTLS_ENABLE).  But it swallows the error, if any.  How is an
> > > > application to tell if ktls is enabled on a particular socket or
> OpenSSL
> > > > session?
> > >
> > > IIRC the lack of answer for this is part of why upstream OpenSSL
> doesn't
> > > have specific KTLS tests enabled in the automated test suite.
> > >
> >
> > getsockopt(_. TCP_TXTLS_ENABLE) returns ENOPROTOOPT.  Is there any reason
> > why it's not implemented?  That might be the easiest way to check for the
> > ktls status of an individual socket.
>
> I think that's probably more of a question for jhb than me.  I don't know
> of a reason why not, but I do know that there is some desire to keep the
> functionality that openssl exposes roughly compatible between linux and
> FreeBSD KTLS.  I don't know whether Linux has something similar.
>
> >
> > >
> > > > * From experiment, I can see that OpenSSL attempts to set
> > > > TCP_TXTLS_ENABLE.  But it doesn't try to set TCP_RXTLS_ENABLE.  Why
> not?
> > > > From reading ktls_start and ossl_statem_server_post_work, it looks
> like
> > > > maybe a single socket cannot have ktls enabled for both sending and
> > > > receiving at the same time.  Is that true?
> > >
> > > No.  They just get enabled separately, since change_cipher_state() is
> > > called separately for read and write transitions.
> > >
> >
> > Apologies if I'm too ignorant, but what is a transition in SSL-speak?
> This
> > is my first attempt at any kind of SSL programming.  What I know from
> > ktrace is that TCP_RXTLS_ENABLE never gets set.
>
> Sorry!  I'm pretty conversant with this stuff (I'm the security area
> director that is responsible for the IETF TLS working group) and don't
> always target the right level.  Basically, for a decent encrypting protocol
> you want different encrytion keys for the read and write direction
> (whichever peer you are), and the TLS (1.3) handshake has a multi-stage key
> hierarchy to try to encrypt as much of it as possible.  So, for example,
> the client will need to update it's encryption key for reading once it
> reads the ServerHello (and before reading the Encrypted Extensions)
> message, even though the keys the client uses for writing don't change at
> that time.  Internally, OpenSSL implements this "transition" of key
> material with a change_cipher_state() abstraction, that takes a flags
> argument (`which`).  The flags indicate which set of keys to update, and
> which direction (read or write).  So, by my read of the code, what's
> *supposed* to happen is that we call:
>
>     if (BIO_set_ktls(bio, &crypto_info, which & SSL3_CC_WRITE))
>
> And if SSL3_CC_WRITE is set, that translates to calling BIO_set_ktls() with
> an `is_txt` value that evaluates to true; otherwise, `is_txt` is false,
> which corresponds to the RX case that you're failing to see happen.
>
> Just to get the boring stuff out of the way: what version of openssl are
> you testing against, and did you verify that OPENSSL_NO_KTLS_RX is not
> defined when ktls_start() is being compiled (so that the setsockopt(fd,
> IPPROTO_TCP, TCP_RXTLS_ENABLE, .) is compiled in at all)?
>
> Thanks,
>
> Ben
>

I'm using the OpenSSL that's in base in 14.0-CURRENT: 1.1.1j-freebsd .  I
haven't recompiled the code to check whether OPENSSL_NO_KTLS_RX is defined,
but it sure looks like it shouldn't be, based on my reading of the source.
-Alan