Re: Getting started with ktls

I'm going to cheat and top post (the discussion looks
pretty convoluted).

- The kernel must be built with "options KERN_TLS"
- OpenSSL must be built with KTLS enabled
- These two sysctls need to be set to 1
   kern.ipc.tls.enable
   kern.ipc.mb_use_ext_pgs

then it all happens "behind the curtain" in the
OpenSSL libraries.

To find out if it is enabled, do the following after
the handshake. (SSL_connect() or SSL_accept() or ??)

        ret = BIO_get_ktls_send(SSL_get_wbio(ssl));
        if (ret != 0)
                ret = BIO_get_ktls_recv(SSL_get_rbio(ssl));
        if (ret != 0)
                /* KTLS enabled for both send and recv. */
The calls return non-zero if it is enabled for that direction.

You'll need it to use TLS 1.2 if you want both directions
to work at this time.
(The code is in usr.sbin/rpc.tlsclntd and usr.sbin/rpc.tlsservd.
 Much easier to look at than man pages imho.)

--> Now you can sosend()/soreceive() on the socket
      in the kernel. If your data is already in mbufs, then
      they must be M_EXTPG mbufs. There is a function
      that copies an mbuf chain into M_EXTPG mbufs,
      but I can't remember what I called it.

rick

________________________________________
From: owner-freebsd-current_at_freebsd.org <owner-freebsd-current_at_freebsd.org> on behalf of Alan Somers <asomers_at_freebsd.org>
Sent: Wednesday, March 10, 2021 10:55 PM
To: Benjamin Kaduk
Cc: FreeBSD CURRENT
Subject: Re: Getting started with ktls

CAUTION: This email originated from outside of the University of Guelph. Do not click links or open attachments unless you recognize the sender and know the content is safe. If in doubt, forward suspicious emails to IThelp_at_uoguelph.ca


On Wed, Mar 10, 2021 at 8:15 PM Benjamin Kaduk <kaduk_at_mit.edu> wrote:

> On Wed, Mar 10, 2021 at 06:17:42PM -0700, Alan Somers wrote:
> > On Wed, Mar 10, 2021 at 5:31 PM Benjamin Kaduk <kaduk_at_mit.edu> wrote:
> >
> > > On Wed, Mar 10, 2021 at 05:18:24PM -0700, Alan Somers wrote:
> > > > I'm trying to make ktls work with "zfs send/recv" to substantially
> reduce
> > > > the CPU utilization of applications like zrepl.  But I have a few
> > > questions:
> > > >
> > > > * ktls(4)'s "Transmit" section says "Once TLS transmit is enabled by
> a
> > > > successful set of the TCP_TXTLS_ENABLE socket option", but the
> "Supported
> > > > Libraries" section says "Applications using a supported library
> should
> > > > generally work with ktls without any changes".  These sentences seem
> to
> > > be
> > > > contradictory.  I think it means that the TCP_TXTLS_ENABLE option is
> > > > necessary, but OpenSSL sets it automatically?
> > >
> > > Yes, OpenSSL sets it automatically for the builtin socket and
> connection
> > > BIO classes.  Applications using other BIO classes will need to do
> things
> > > manually (or implement the appropriate _ctrl() parameters for their BIO
> > > class).
> > >
> > > > * When using OpenSSL, the library will automatically call
> setsockopt(_,
> > > > TCP_TXTLS_ENABLE).  But it swallows the error, if any.  How is an
> > > > application to tell if ktls is enabled on a particular socket or
> OpenSSL
> > > > session?
> > >
> > > IIRC the lack of answer for this is part of why upstream OpenSSL
> doesn't
> > > have specific KTLS tests enabled in the automated test suite.
> > >
> >
> > getsockopt(_. TCP_TXTLS_ENABLE) returns ENOPROTOOPT.  Is there any reason
> > why it's not implemented?  That might be the easiest way to check for the
> > ktls status of an individual socket.
>
> I think that's probably more of a question for jhb than me.  I don't know
> of a reason why not, but I do know that there is some desire to keep the
> functionality that openssl exposes roughly compatible between linux and
> FreeBSD KTLS.  I don't know whether Linux has something similar.
>
> >
> > >
> > > > * From experiment, I can see that OpenSSL attempts to set
> > > > TCP_TXTLS_ENABLE.  But it doesn't try to set TCP_RXTLS_ENABLE.  Why
> not?
> > > > From reading ktls_start and ossl_statem_server_post_work, it looks
> like
> > > > maybe a single socket cannot have ktls enabled for both sending and
> > > > receiving at the same time.  Is that true?
> > >
> > > No.  They just get enabled separately, since change_cipher_state() is
> > > called separately for read and write transitions.
> > >
> >
> > Apologies if I'm too ignorant, but what is a transition in SSL-speak?
> This
> > is my first attempt at any kind of SSL programming.  What I know from
> > ktrace is that TCP_RXTLS_ENABLE never gets set.
>
> Sorry!  I'm pretty conversant with this stuff (I'm the security area
> director that is responsible for the IETF TLS working group) and don't
> always target the right level.  Basically, for a decent encrypting protocol
> you want different encrytion keys for the read and write direction
> (whichever peer you are), and the TLS (1.3) handshake has a multi-stage key
> hierarchy to try to encrypt as much of it as possible.  So, for example,
> the client will need to update it's encryption key for reading once it
> reads the ServerHello (and before reading the Encrypted Extensions)
> message, even though the keys the client uses for writing don't change at
> that time.  Internally, OpenSSL implements this "transition" of key
> material with a change_cipher_state() abstraction, that takes a flags
> argument (`which`).  The flags indicate which set of keys to update, and
> which direction (read or write).  So, by my read of the code, what's
> *supposed* to happen is that we call:
>
>     if (BIO_set_ktls(bio, &crypto_info, which & SSL3_CC_WRITE))
>
> And if SSL3_CC_WRITE is set, that translates to calling BIO_set_ktls() with
> an `is_txt` value that evaluates to true; otherwise, `is_txt` is false,
> which corresponds to the RX case that you're failing to see happen.
>
> Just to get the boring stuff out of the way: what version of openssl are
> you testing against, and did you verify that OPENSSL_NO_KTLS_RX is not
> defined when ktls_start() is being compiled (so that the setsockopt(fd,
> IPPROTO_TCP, TCP_RXTLS_ENABLE, .) is compiled in at all)?
>
> Thanks,
>
> Ben
>

I'm using the OpenSSL that's in base in 14.0-CURRENT: 1.1.1j-freebsd .  I
haven't recompiled the code to check whether OPENSSL_NO_KTLS_RX is defined,
but it sure looks like it shouldn't be, based on my reading of the source.
-Alan
_______________________________________________
freebsd-current_at_freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscribe_at_freebsd.org"