Re: OpenSSH HPN

On 2015-11-14 02:47, John-Mark Gurney wrote:
> Allan Jude wrote this message on Thu, Nov 12, 2015 at 17:57 -0500:
>> On 2015-11-12 12:56, John-Mark Gurney wrote:
>>> Allan Jude wrote this message on Thu, Nov 12, 2015 at 12:15 -0500:
>>>> On 2015-11-11 19:06, Slawa Olhovchenkov wrote:
>>>>> On Wed, Nov 11, 2015 at 01:32:27PM -0800, Bryan Drewery wrote:
>>>>>
>>>>>> On 11/10/2015 1:42 AM, Dag-Erling Smørgrav wrote:
>>>>>>>  I would also like to remove the NONE cipher
>>>>>>> patch, which is also available in the port (off by default, just like in
>>>>>>> base).
>>>>>>
>>>>>> Fun fact, it's been broken in the port for several months with no
>>>>>> complaints. It was just reported and fixed upstream in the last day and
>>>>>> I wrote in a similar fix in the port. That speaks a lot about its usage
>>>>>> in the port currently.
>>>>>
>>>>> I am try using NPH/NONE with base ssh and confused: don't see
>>>>> performance rise, too complex to enable and too complex for use.
>>>>
>>>> I did a few quick (and dirty) benchmarks and it shows that the NONE
>>>> cipher definitely makes a difference. Version of OpenSSL also seems to
>>>> make a difference, as one might expect.
>>>>
>>>> Note: openssh from ports seems to link against both base and ports
>>>> libcrypto, I am still trying to make sure this isn't corrupting my
>>>> benchmark results.
>>>
>>> You don't need the aesni.ko module loaded for OpenSSL (which is how
>>> OpenSSH uses most crypto algos) to use AES-NI..
>>>
>>> Also, do you set any sysctl's to play w/ the buffer sizes or anything?
>>>
>>>> I am still debugging my dummynet setup to be able to prove that HPN
>>>> makes a difference (but it does).
>>>
>>> Does my example on the page not work for you?
>>>
>>>> https://wiki.freebsd.org/SSHPerf
>>>
>>
>> I found that when I set even 5ms of delay with dummynet, bandwidth over
>> the LAN drops more than it should. Dummynet is limiting the rate rather
>> than just adding the delay. I am investigating.
>>
>> I found this document:
>> http://www.cs.unc.edu/~jeffay/dirt/FAQ/hstcp-howto.pdf
>>
>> Which is from the 6.x era, but suggests:
>>
>> "One subtle bug exists in the stock Dummynet implementation that should
>> be corrected for experiments.  When a packet arrives in dummynet it is
>> shoved into a queue which limits the bandwidth a TCP flow may use. Upon
>> exit from the queue, the packet is transferred to a pipe where it sits
>> for any configured amount of delay time and might possibly be dropped
>> depending on the loss probability. Once the delay time has passed, the
>> packet is released to ip output."
>>
>> May be the cause of my problem
> 
> Ahhh, probably need to adjust:
> net.inet.ip.dummynet.pipe_byte_limit: 1048576
> net.inet.ip.dummynet.pipe_slot_limit: 100
> 
> But even w/ the above limits and 5ms, you should still be able to push
> 200MB/sec...
> 

I worked with Hiren and some of his dtrace magic and figured out that
dummynet was not my issue. I didn't end up needing to change the
dummynet pipe slot/byte limit in order to get the full 10gig/sec even
with 100ms delay from dummynet.

You just need to adjust:

net.inet.tcp.sendbuf_max=BDP
net.inet.tcp.recvbuf_max=BDP

kern.ipc.maxsockbuf= ( BDP * (2048+256) ) / 2048

for a 50ms RTT:

BDP = 10gbps * .05 = ~60mb


It can also greatly help to increase:
net.inet.tcp.abc_l_var

Which is how many additional segments the CWNDs is incremented by each
RTT during slow-start.


I am still working on my set of benchmarks to show what different HPN
makes with different RTT values, as well as what might be required to
achieve max throughput for SSH over both LAN and the internet.

(For my company, we regularly transmit 500GB ZFS datasets over the
public internet on 1gbps or 2x1gbps connections)

-- 
Allan Jude