Re: using ConnectX card as Ethernet (mlxen)

From: Eggert, Lars <lars_at_netapp.com>
Date: Tue, 21 Jan 2014 09:04:46 +0000
Hi,

On 2014-1-20, at 21:59, John Baldwin <jhb_at_freebsd.org> wrote:
> I believe this should work, yes.  Getting a crashdump or the panic messages 
> would be really helpful in figuring out why it isn't.  Thanks.

I rebuilt the kernel, and see no crashes anymore. So that's good.

But there are a bunch of other issues that maybe someone has some ideas about:


(1) Late attach

The ConnectX-3 attaches very late during the boot process, after the system is already in single-user mode. See the attached dmesg; pci17 and pci18 (there are two identical cards in this system) first show as "no driver attached" during the PCI bus enumeration. Only after the system is single-user mode does the mlx4_core attach to the cards.

That means that e.g. trying to set sysctls for these cards in /etc/sysctl.conf, or configuring their IP addresses via rc.conf is not possible. At the moment, I work around this by sleeping in rc.local and then doing assignments there, but that's a hack.

Any clues why these cards attach so late?


(2) Device numbers change

After booting, these cards show up in InfiniBand mode:

ib0: flags=8002<BROADCAST,MULTICAST> metric 0 mtu 65520
	options=80018<VLAN_MTU,VLAN_HWTAGGING,LINKSTATE>
	lladdr 80.0.0.48.fe.80.0.0.0.0.0.0.f4.52.14.3.0.10.d1.21
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
ib1: flags=8002<BROADCAST,MULTICAST> metric 0 mtu 65520
	options=80018<VLAN_MTU,VLAN_HWTAGGING,LINKSTATE>
	lladdr 80.0.0.49.fe.80.0.0.0.0.0.0.f4.52.14.3.0.10.d1.22
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
ib2: flags=8002<BROADCAST,MULTICAST> metric 0 mtu 65520
	options=80018<VLAN_MTU,VLAN_HWTAGGING,LINKSTATE>
	lladdr 80.0.0.48.fe.80.0.0.0.0.0.0.f4.52.14.3.0.10.d0.d1
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
ib3: flags=8002<BROADCAST,MULTICAST> metric 0 mtu 65520
	options=80018<VLAN_MTU,VLAN_HWTAGGING,LINKSTATE>
	lladdr 80.0.0.49.fe.80.0.0.0.0.0.0.f4.52.14.3.0.10.d0.d2
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

Then I force one into Ethernet mode:

# sysctl sys.device.mlx4_core0.mlx4_port1=eth
sys.device.mlx4_core0.mlx4_port1: auto (ib) -> eth

and the device numbers on the ib devices change: ib1 is now ib4, and I have a new mlxen0 device.

ib2: flags=8002<BROADCAST,MULTICAST> metric 0 mtu 65520
	options=80018<VLAN_MTU,VLAN_HWTAGGING,LINKSTATE>
	lladdr 80.0.0.48.fe.80.0.0.0.0.0.0.f4.52.14.3.0.10.d0.d1
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
ib3: flags=8002<BROADCAST,MULTICAST> metric 0 mtu 65520
	options=80018<VLAN_MTU,VLAN_HWTAGGING,LINKSTATE>
	lladdr 80.0.0.49.fe.80.0.0.0.0.0.0.f4.52.14.3.0.10.d0.d2
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
mlxen0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=d05bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,LRO,VLAN_HWFILTER,VLAN_HWTSO,LINKSTATE>
	ether f4:52:14:10:d1:21
	inet6 fe80::f652:14ff:fe10:d121%mlxen0 prefixlen 64 scopeid 0xe 
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	media: Ethernet autoselect
	status: no carrier
ib4: flags=8002<BROADCAST,MULTICAST> metric 0 mtu 65520
	options=80018<VLAN_MTU,VLAN_HWTAGGING,LINKSTATE>
	lladdr 80.0.0.4a.fe.80.0.0.0.0.0.0.f4.52.14.3.0.10.d1.22
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

When I change another port into Ethernet mode

# sysctl sys.device.mlx4_core0.mlx4_port2=eth
sys.device.mlx4_core0.mlx4_port2: auto (ib) -> eth

device numbers change again. Now mxlen0 disappears and becomes mxlen1, and I have a new mxlen2 device:

ib2: flags=8002<BROADCAST,MULTICAST> metric 0 mtu 65520
	options=80018<VLAN_MTU,VLAN_HWTAGGING,LINKSTATE>
	lladdr 80.0.0.48.fe.80.0.0.0.0.0.0.f4.52.14.3.0.10.d0.d1
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
ib3: flags=8002<BROADCAST,MULTICAST> metric 0 mtu 65520
	options=80018<VLAN_MTU,VLAN_HWTAGGING,LINKSTATE>
	lladdr 80.0.0.49.fe.80.0.0.0.0.0.0.f4.52.14.3.0.10.d0.d2
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
mlxen1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=d05bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,LRO,VLAN_HWFILTER,VLAN_HWTSO,LINKSTATE>
	ether f4:52:14:10:d1:21
	inet6 fe80::f652:14ff:fe10:d121%mlxen1 prefixlen 64 scopeid 0xe 
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	media: Ethernet autoselect
	status: no carrier
mlxen2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=d05bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,LRO,VLAN_HWFILTER,VLAN_HWTSO,LINKSTATE>
	ether f4:52:14:10:d1:22
	inet6 fe80::f652:14ff:fe10:d122%mlxen2 prefixlen 64 scopeid 0xf 
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	media: Ethernet autoselect
	status: no carrier

Changing the other two ports (on the second card) to Ethernet mode 

# sysctl sys.device.mlx4_core1.mlx4_port1=eth
sys.device.mlx4_core1.mlx4_port1: auto (ib) -> eth
# sysctl sys.device.mlx4_core1.mlx4_port2=eth
sys.device.mlx4_core1.mlx4_port2: auto (ib) -> eth

leaves me with mlxen1, mlxen2, mlxen4 and mlxen 5:

mlxen1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=d05bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,LRO,VLAN_HWFILTER,VLAN_HWTSO,LINKSTATE>
	ether f4:52:14:10:d1:21
	inet6 fe80::f652:14ff:fe10:d121%mlxen1 prefixlen 64 scopeid 0xe 
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	media: Ethernet autoselect
	status: no carrier
mlxen2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=d05bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,LRO,VLAN_HWFILTER,VLAN_HWTSO,LINKSTATE>
	ether f4:52:14:10:d1:22
	inet6 fe80::f652:14ff:fe10:d122%mlxen2 prefixlen 64 scopeid 0xf 
	inet 0.0.0.0 netmask 0xff000000 broadcast 255.255.255.255 
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	media: Ethernet autoselect (40Gbase-CR4 <full-duplex,rxpause,txpause>)
	status: active
mlxen4: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=d05bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,LRO,VLAN_HWFILTER,VLAN_HWTSO,LINKSTATE>
	ether f4:52:14:10:d0:d1
	inet6 fe80::f652:14ff:fe10:d0d1%mlxen4 prefixlen 64 scopeid 0x10 
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	media: Ethernet autoselect
	status: no carrier
mlxen5: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=d05bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,LRO,VLAN_HWFILTER,VLAN_HWTSO,LINKSTATE>
	ether f4:52:14:10:d0:d2
	inet6 fe80::f652:14ff:fe10:d0d2%mlxen5 prefixlen 64 scopeid 0x11 
	inet 0.0.0.0 netmask 0xff000000 broadcast 255.255.255.255 
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	media: Ethernet autoselect (10Gbase-CX4 <full-duplex,rxpause,txpause>)
	status: active

Needless to say, having devices change numbers is problematic.


(3) 40G TCP performance

I barely get over 10G with netperf over the 40G interfaces:

root_at_one:~ # netperf -H two-mlxen2 -- -s512k -S512K
MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to two-mlxen2.muclab () port 0 AF_INET : histogram : interval : dirty data : demo
Recv   Send    Send                          
Socket Socket  Message  Elapsed              
Size   Size    Size     Time     Throughput  
bytes  bytes   bytes    secs.    10^6bits/sec  

524288 512000 512000    10.07    10268.01   

Any clues as to what could be limiting performance here?

Thanks,
Lars



Received on Tue Jan 21 2014 - 08:04:54 UTC

This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:40:46 UTC