(unknown charset) Re: HEADS-UP: IFLIB implementations of sys/dev/e1000 em, lem, igb pending

On Tue, 24 Jan 2017, Sean Bruno wrote:

> On 01/24/17 08:27, Olivier Cochard-Labbé wrote:
>> On Tue, Jan 24, 2017 at 3:17 PM, Sean Bruno <sbruno_at_freebsd.org
>> <mailto:sbruno_at_freebsd.org>> wrote:
>>
>>     Did you increase the number of rx/tx rings to 8 and the number of
>>     descriptors to 4k in your tests or just the defaults?
>>
>> Tuning are same as described in my previous email (rxd|txd=2048, rx|tx
>> process_limit=-1, max_interrupt_rate=16000).
>> [root_at_apu2]~# sysctl hw.igb.
>> hw.igb.tx_process_limit: -1
>> hw.igb.rx_process_limit: -1
>> hw.igb.num_queues: 0
>> hw.igb.header_split: 0
>> hw.igb.max_interrupt_rate: 16000
>> hw.igb.enable_msix: 1
>> hw.igb.enable_aim: 1
>> hw.igb.txd: 2048
>> hw.igb.rxd: 2048
>
> Oh, I think you missed my note on these.  In order to adjust txd/rxd you
> need to tweak the iflib version of these numbers.  nrxds/ntxds should be
> adjust upwards to your value of 2048.  nrxqs/ntxqs should be adjust
> upwards to 8, I think, so you can test equivalent settings to the legacy
> driver.
>
> Specifically, you may want to adjust these:
>
> dev.em.0.iflib.override_nrxds: 0
> dev.em.0.iflib.override_ntxds: 0
>
> dev.em.0.iflib.override_nrxqs: 0
> dev.em.0.iflib.override_ntxqs: 0

That is painful.

My hack to increase the ifq length also no longer works:

X Index: if_em.c
X ===================================================================
X --- if_em.c	(revision 312696)
X +++ if_em.c	(working copy)
X _at__at_ -1,3 +1,5 _at__at_
X +int em_qlenadj = -1;
X +

-1 gives a null adjustment; 0 gives a default (very large ifq), and other
values give a non-null adustment.

X  /*-
X   * Copyright (c) 2016 Matt Macy <mmacy_at_nextbsd.org>
X   * All rights reserved.
X _at__at_ -2488,7 +2490,10 _at__at_
X 
X  	/* Single Queue */
X          if (adapter->tx_num_queues == 1) {
X -	  if_setsendqlen(ifp, scctx->isc_ntxd[0] - 1);
X +	  if (em_qlenadj == 0)
X +	    em_qlenadj = imax(2 * tick, 0) * 15 / 10;
X +	    // lem_qlenadj = imax(2 * tick, 0) * 42 / 100;
X +	  if_setsendqlen(ifp, scctx->isc_ntxd[0] + em_qlenadj);
X  	  if_setsendqready(ifp);
X  	}
X

I don't want larger hardware queues, but sometimes want larger software
queues.  ifq's used to give them.  The if_setsenqlen() call is still there.
but no longer gives them.

The large queues are needed for backet blasting benchmarks since select()
doesn't work for udp sockets, so if the queues fill up then the benchmarks
must busy-wait or sleep waiting for them to drain, and timeout granularity
tends to prevent short sleeps from working so the queues run dry while
sleeping unless the queues are very large.

Bruce