Re: [RFC] Patch to add Software/Generic Segmentation Offload (GSO) support in FreeBSD

From: Stefano Garzarella <stefanogarzarella_at_gmail.com>
Date: Wed, 17 Sep 2014 20:18:44 +0200
Hi Adrian,
the results that I sent, regard just one flow, but I can try with two
simultaneous flows and I'll send you the results.

Thanks,
Stefano

2014-09-17 19:27 GMT+02:00 Adrian Chadd <adrian_at_freebsd.org>:

> Hi!
>
> Cool!
>
> How many flows were you testing with? Just one or two?
>
> It's for outbound, so it's not _as_ big a deal as it is for inbound,
> but it'd still be nice to know.
>
>
> -a
>
>
> On 17 September 2014 01:27, Stefano Garzarella
> <stefanogarzarella_at_gmail.com> wrote:
> > Hi all,
> > I have recently worked, during my master’s thesis with the supervision
> > of Prof. Luigi Rizzo, on a project to add GSO (Generic Segmentation
> > Offload) support in FreeBSD. I will present this project at EuroBSDcon
> > 2014, in Sofia (Bulgaria) on September 28, 2014.
> >
> > Following is a brief description of our project:
> >
> > The use of large frames makes network communication much less
> > demanding for the CPU. Yet, backward compatibility and slow links
> > requires the use of 1500 byte or smaller frames.  Modern NICs with
> > hardware TCP segmentation offloading (TSO) address this problem.
> > However, a generic software version (GSO) provided by the OS has
> > reason to exist, for use on paths with no suitable hardware, such
> > as between virtual machines or with older or buggy NICs.
> >
> > Much of the advantage of TSO comes from crossing the network stack only
> > once per (large) segment instead of once per 1500-byte frame.
> > GSO does the same both for segmentation (TCP) and fragmentation (UDP)
> > by doing these operations as late as possible. Ideally, this could be
> done
> > within the device driver, but that would require modifications to all
> > drivers.
> > A more convenient, similarly effective approach is to segment
> > just before the packet is passed to the driver (in ether_output()).
> >
> > Our preliminary implementation supports TCP and UDP on IPv4/IPv6;
> > it only intercepts packets large than the MTU (others are left
> unchanged),
> > and only when GSO is marked as enabled for the interface.
> >
> > Segments larger than the MTU are not split in tcp_output(),
> > udp_output(), or ip_output(), but marked with a flag (contained in
> > m_pkthdr.csum_flags), which is processed by ether_output() just
> > before calling the device driver.
> >
> > ether_output(), through gso_dispatch(), splits the large frame as needed,
> > creating headers and possibly doing checksums if not supported by
> > the hardware.
> >
> > In experiments agains an LRO-enabled receiver (otherwise TSO/GSO
> > are ineffective) we have seen the following performance,
> > taken at different clock speeds (because at top speeds the
> > 10G link becomes the bottleneck):
> >
> >
> >     Testing enviroment (all with Intel 10Gbit NIC)
> >     Sender: FreeBSD 11-CURRENT - CPU i7-870 at 2.93 GHz + Turboboost
> >     Receiver: Linux 3.12.8 - CPU i7-3770K at 3.50GHz + Turboboost
> >     Benchmark tool: netperf 2.6.0
> >
> >     --- TCP/IPv4 packets (checksum offloading enabled) ---
> >     Freq.      TSO       GSO     none     Speedup
> >     [GHz]     [Gbps]   [Gbps]   [Gbps]   GSO-none
> >     2.93       9347      9298      8308     12 %
> >     2.53       9266      9401      6771     39 %
> >     2.00       9408      9294      5499     69 %
> >     1.46       9408      8087      4075     98 %
> >     1.05       9408      5673      2884     97 %
> >     0.45       6760      2206      1244     77 %
> >
> >
> >     --- TCP/IPv6 packets (checksum offloading enabled) ---
> >     Freq.      TSO       GSO     none     Speedup
> >     [GHz]     [Gbps]   [Gbps]   [Gbps]   GSO-none
> >     2.93       7530      6939      4966     40 %
> >     2.53       5133      7145      4008     78 %
> >     2.00       5965      6331      3152    101 %
> >     1.46       5565      5180      2348    121 %
> >     1.05       8501      3607      1732    108 %
> >     0.45       3665      1505        651    131 %
> >
> >
> >     --- UDP/IPv4 packets (9K) ---
> >     Freq.      GSO      none     Speedup
> >     [GHz]     [Gbps]   [Gbps]   GSO-none
> >     2.93       9440      8084     17 %
> >     2.53       7772      6649     17 %
> >     2.00       6336      5338     19 %
> >     1.46       4748      4014     18 %
> >     1.05       3359      2831     19 %
> >     0.45       1312      1120     17 %
> >
> >
> >     --- UDP/IPv6 packets (9K) ---
> >     Freq.      GSO      none     Speedup
> >     [GHz]     [Gbps]   [Gbps]   GSO-none
> >     2.93       7281      6197     18 %
> >     2.53       5953      5020     19 %
> >     2.00       4804      4048     19 %
> >     1.46       3582      3004     19 %
> >     1.05       2512      2092     20 %
> >     0.45         998        826     21 %
> >
> > We tried to change as little as possible the network stack to add
> > GSO support. To avoid changing API/ABI, we temporarily used spare
> > fields in struct tcpcb (TCP Control Block) and struct ifnet to store
> > some information related to GSO (enabled, max burst size, etc.).
> > The code that performs the segmentation/fragmentation is contained
> > in the file gso.[h|c] in sys/net.  We used 4 bit in m_pkthdr.csum_flags
> > (CSUM_GSO_MASK) to encode the packet type (TCP/IPv4, TCP/IPv6, etc)
> > to prevent access to the TCP/IP/Ethernet headers of each packet.
> > In ether_output_frame(), if the packet requires the GSO
> > ((m->m_pkthdr.csum_flags & CSUM_GSO_MASK) != 0), it is segmented
> > or fragmented, and then they are sent to the device driver.
> >
> > At https://github.com/stefano-garzarella/freebsd-gso
> > you can find the kernel patches for FreeBSD-current, FreeBSD
> > 10-stable, FreeBSD 9-stable, a simple application (gso-stats.c)
> > that prints the GSO statistics and picobsd images with GSO support.
> >
> > At https://github.com/stefano-garzarella/freebsd-gso-src
> > you can get the FreeBSD source with GSO patch (various branch for
> > FreeBSD current, 10-stable, 9-stable).
> >
> > Any feedbacks, comments, questions are welcome.
> >
> > Thank you very much,
> > Stefano Garzarella
> >
> >
> ------------------------------------------------------------------------------------------------------------
> > How to use GSO:
> >
> > - Apply the right kernel patch.
> >
> > - To compile the GSO support add ‘ options GSO ' to your kernel config
> file
> > and
> >    rebuild a kernel.
> >
> > - To manage the GSO parameters there are some sysctls:
> >      - net.inet.tcp.gso - GSO enable on TCP communications (!=0)
> >      - net.inet.udp.gso - GSO enable on UDP communications (!=0)
> >
> >      - for each interface:
> >           - net.gso.dev."ifname”.max_burst - GSO burst length limit
> >                 [default: IP_MAXPACKET=65535]
> >           - net.gso.dev."ifname”.enable_gso - GSO enable on “ifname”
> > interface (!=0)
> >
> > - To show statistics:
> >      - make sure that the GSO_STATS macro is defined in sys/net/gso.h
> >      - use the simple gso-stats.c application to access the sysctl
> > net.gso.stats
> >        that contains the address of the gsostats structure (defined in
> > gso.h)
> >        which records the statistics.  (compile with
> > -I/path/to/kernel/src/patched/)
> >
> ------------------------------------------------------------------------------------------------------------
> >
> > --
> > *Stefano Garzarella*
> > stefano.garzarella_at_gmail.com
> > _______________________________________________
> > freebsd-current_at_freebsd.org mailing list
> > http://lists.freebsd.org/mailman/listinfo/freebsd-current
> > To unsubscribe, send any mail to "
> freebsd-current-unsubscribe_at_freebsd.org"
>



-- 
*Stefano Garzarella*
stefano.garzarella_at_gmail.com
Received on Wed Sep 17 2014 - 16:18:45 UTC

This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:40:52 UTC