Re: using ConnectX card as Ethernet (mlxen)

From: Eggert, Lars <lars_at_netapp.com>
Date: Tue, 21 Jan 2014 09:05:48 +0000
On 2014-1-21, at 10:04, Lars Eggert <lars_at_netapp.com> wrote:
> See the attached dmesg

which I of course forget to attach (sigh). See below.

Lars


GDB: no debug ports present970
KDB: debugger backends: ddb
KDB: current backend: ddb
Copyright (c) 1992-2014 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights
reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 11.0-CURRENT #8 ab08c30(fas3270)-dirty: Tue Jan 21 09:07:36 CET
2014

elars_at_stanley.muccbc.hq.netapp.com:/usr/home/elars/obj/usr/home/elars/src/sys/FAS3270
amd64
FreeBSD clang version 3.3 (tags/RELEASE_33/final 183502) 20130610
CPU: Intel(R) Xeon(R) CPU           E5240  _at_ 3.00GHz (3000.17-MHz
K8-class CPU)
  Origin="GenuineIntel"  Id=0x1067a  Family=0x6  Model=0x17  Stepping=10

Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>

Features2=0xc0ce3bd<SSE3,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,SSE4.1,XSAVE,OSXSAVE>
  AMD Features=0x20100800<SYSCALL,NX,LM>
  AMD Features2=0x1<LAHF>
  TSC: P-state invariant, performance statistics
real memory  = 18253611008 (17408 MB)
avail memory = 16599695360 (15830 MB)
MPTable: <NETAPP   SB_XVI      >
Event timer "LAPIC" quality 400
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
FreeBSD/SMP: 2 package(s) x 2 core(s)
 cpu0 (BSP): APIC ID:  0
 cpu1 (AP): APIC ID:  1
 cpu2 (AP): APIC ID:  6
 cpu3 (AP): APIC ID:  7
random device not loaded; using insecure entropy
ioapic0: Assuming intbase of 0
ioapic0 <Version 2.0> irqs 0-23 on motherboard
netmap: loaded module
random: <Software, Yarrow> initialized
smbios0: <System Management BIOS> at iomem 0xf6c00-0xf6c1e on
motherboard
smbios0: Version: 2.5
cryptosoft0: <software crypto> on motherboard
pcib0: <MPTable Host-PCI bridge> pcibus 0 on motherboard
pci0: <PCI bus> on pcib0
pcib1: <MPTable PCI-PCI bridge> at device 2.0 on pci0
pci1: <PCI bus> on pcib1
cxgbc0: <Carnegie T3 onboard SR KR, 2 ports> mem
0xdd001000-0xdd001fff,0xdc800000-0xdcffffff,0xdd000000-0xdd000fff irq 16
at device 0.0 on pci1
cxgbc0: AD8158 0xf=0x3 0x1=0xf
cxgbc0: using MSI-X interrupts (9 vectors)
cxgb0: <Port 0 10GBASE-R> on cxgbc0
cxgb0: Ethernet address: 00:a0:98:30:c2:2a
cxgb1: <Port 1 10GBASE-R> on cxgbc0
cxgb1: Ethernet address: 00:a0:98:30:c2:2b
cxgbc0: Firmware Version 7.11.0
pcib2: <PCI-PCI bridge> at device 3.0 on pci0
pci2: <PCI bus> on pcib2
pcib3: <MPTable PCI-PCI bridge> at device 4.0 on pci0
pci3: <PCI bus> on pcib3
pcib4: <PCI-PCI bridge> mem 0xdd300000-0xdd31ffff irq 16 at device 0.0
on pci3
pci4: <PCI bus> on pcib4
pcib3: unable to route slot 0 INTB
pcib5: <PCI-PCI bridge> irq 16 at device 4.0 on pci4
pci5: <PCI bus> on pcib5
pcib6: <MPTable PCI-PCI bridge> irq 10 at device 5.0 on pci4
pci6: <PCI bus> on pcib6
ix0: <Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 2.5.15>
mem 0xdd400000-0xdd47ffff,0xdd500000-0xdd503fff irq 17 at device 0.0 on
pci6
ix0: Using MSIX interrupts with 5 vectors
ix0: Ethernet address: 90:e2:ba:37:d5:b4
ix0: PCI Express Bus: Speed 5.0GT/s Width x8
001.000008 [2141] netmap_attach             success for ix0
ix1: <Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 2.5.15>
mem 0xdd480000-0xdd4fffff,0xdd504000-0xdd507fff irq 18 at device 0.1 on
pci6
ix1: Using MSIX interrupts with 5 vectors
ix1: Ethernet address: 90:e2:ba:37:d5:b5
ix1: PCI Express Bus: Speed 5.0GT/s Width x8
001.000009 [2141] netmap_attach             success for ix1
pcib7: <PCI-PCI bridge> irq 16 at device 8.0 on pci4
pci7: <PCI bus> on pcib7
pcib8: <PCI-PCI bridge> at device 0.0 on pci7
pci8: <PCI bus> on pcib8
pcib9: <MPTable PCI-PCI bridge> at device 0.0 on pci8
pci9: <PCI bus> on pcib9
em0: <Intel(R) PRO/1000 Network Connection 7.3.8> mem
0xdd620000-0xdd63ffff,0xdd600000-0xdd61ffff irq 16 at device 0.0 on pci9
em0: Using an MSI interrupt
em0: Ethernet address: 00:1b:21:a8:a5:34
001.000010 [2141] netmap_attach             success for em0
em1: <Intel(R) PRO/1000 Network Connection 7.3.8> mem
0xdd660000-0xdd67ffff,0xdd640000-0xdd65ffff irq 17 at device 0.1 on pci9
em1: Using an MSI interrupt
em1: Ethernet address: 00:1b:21:a8:a5:35
001.000011 [2141] netmap_attach             success for em1
pcib10: <MPTable PCI-PCI bridge> at device 1.0 on pci8
pci10: <PCI bus> on pcib10
em2: <Intel(R) PRO/1000 Network Connection 7.3.8> mem
0xdd720000-0xdd73ffff,0xdd700000-0xdd71ffff irq 17 at device 0.0 on
pci10
em2: Using an MSI interrupt
em2: Ethernet address: 00:1b:21:a8:a5:36
001.000012 [2141] netmap_attach             success for em2
em3: <Intel(R) PRO/1000 Network Connection 7.3.8> mem
0xdd760000-0xdd77ffff,0xdd740000-0xdd75ffff irq 18 at device 0.1 on
pci10
em3: Using an MSI interrupt
em3: Ethernet address: 00:1b:21:a8:a5:37
001.000013 [2141] netmap_attach             success for em3
pcib11: <PCI-PCI bridge> at device 5.0 on pci0
pci11: <PCI bus> on pcib11
pcib12: <PCI-PCI bridge> at device 6.0 on pci0
pci12: <PCI bus> on pcib12
pcib0: unable to route slot 6 INTA
pcib13: <MPTable PCI-PCI bridge> mem 0xddd00000-0xddd1ffff irq 5 at
device 0.0 on pci12
pci13: <PCI bus> on pcib13
pcib13: unable to route slot 4 INTA
pcib13: unable to route slot 5 INTA
pcib13: unable to route slot 6 INTA
pcib14: <MPTable PCI-PCI bridge> irq 16 at device 0.0 on pci13
pci14: <PCI bus> on pcib14
pcib15: <PCI-PCI bridge> mem 0xdda00000-0xdda1ffff irq 16 at device 0.0
on pci14
pci15: <PCI bus> on pcib15
pcib14: unable to route slot 0 INTB
pcib14: unable to route slot 0 INTB
pcib14: unable to route slot 0 INTB
pcib16: <PCI-PCI bridge> irq 10 at device 1.0 on pci15
pci16: <PCI bus> on pcib16
pcib17: <MPTable PCI-PCI bridge> irq 16 at device 4.0 on pci15
pci17: <PCI bus> on pcib17
pci17: <network> at device 0.0 (no driver attached)
pcib18: <MPTable PCI-PCI bridge> irq 10 at device 5.0 on pci15
pci18: <PCI bus> on pcib18
pci18: <network> at device 0.0 (no driver attached)
pcib19: <PCI-PCI bridge> irq 16 at device 8.0 on pci15
pci19: <PCI bus> on pcib19
pcib20: <PCI-PCI bridge> irq 10 at device 9.0 on pci15
pci20: <PCI bus> on pcib20
pcib21: <PCI-PCI bridge> irq 5 at device 4.0 on pci13
pci21: <PCI bus> on pcib21
pcib22: <MPTable PCI-PCI bridge> irq 10 at device 5.0 on pci13
pci22: <PCI bus> on pcib22
pci22: <serial bus, Fibre Channel> at device 0.0 (no driver attached)
pci22: <serial bus, Fibre Channel> at device 0.1 (no driver attached)
pcib23: <MPTable PCI-PCI bridge> irq 11 at device 6.0 on pci13
pci23: <PCI bus> on pcib23
pci23: <mass storage, SAS> at device 0.0 (no driver attached)
pcib24: <PCI-PCI bridge> at device 7.0 on pci0
pci24: <PCI bus> on pcib24
pci0: <base peripheral> at device 8.0 (no driver attached)
uhci0: <Intel 82801I (ICH9) USB controller> port 0x1820-0x183f irq 16 at
device 26.0 on pci0
usbus0 on uhci0
ehci0: <Intel 82801I (ICH9) USB 2.0 controller> mem
0xde101000-0xde1013ff irq 18 at device 26.7 on pci0
usbus1: EHCI version 1.0
usbus1 on ehci0
pcib25: <MPTable PCI-PCI bridge> irq 16 at device 28.0 on pci0
pci25: <PCI bus> on pcib25
em4: <Intel(R) PRO/1000 Network Connection 7.3.8> port 0x3000-0x301f mem
0xdde20000-0xdde3ffff,0xdde00000-0xdde1ffff irq 16 at device 0.0 on
pci25
em4: Using an MSI interrupt
em4: Ethernet address: 00:a0:98:30:c2:28
001.000015 [2141] netmap_attach             success for em4
em5: <Intel(R) PRO/1000 Network Connection 7.3.8> port 0x3020-0x303f mem
0xdde60000-0xdde7ffff,0xdde40000-0xdde5ffff irq 17 at device 0.1 on
pci25
em5: Using an MSI interrupt
em5: Ethernet address: 00:a0:98:30:c2:29
001.000016 [2141] netmap_attach             success for em5
pcib26: <MPTable PCI-PCI bridge> irq 16 at device 28.4 on pci0
pci26: <PCI bus> on pcib26
bge0: <Broadcom NetXtreme Gigabit Ethernet Controller, ASIC rev.
0x004201> mem 0xddf00000-0xddf0ffff irq 16 at device 0.0 on pci26
bge0: CHIP ID 0x00004201; ASIC REV 0x04; CHIP REV 0x42; PCI-E
miibus0: <MII bus> on bge0
brgphy0: <BCM5750 1000BASE-T media interface> PHY 1 on miibus0
brgphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT,
1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow
bge0: Ethernet address: 00:a0:98:30:c2:2c
pcib27: <MPTable PCI-PCI bridge> irq 17 at device 28.5 on pci0
pci27: <PCI bus> on pcib27
bge1: <Broadcom NetXtreme Gigabit Ethernet Controller, ASIC rev.
0x004201> mem 0xde000000-0xde00ffff irq 17 at device 0.0 on pci27
bge1: CHIP ID 0x00004201; ASIC REV 0x04; CHIP REV 0x42; PCI-E
miibus1: <MII bus> on bge1
brgphy1: <BCM5750 1000BASE-T media interface> PHY 1 on miibus1
brgphy1:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT,
1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow
bge1: Ethernet address: 00:a0:98:30:c2:2d
uhci1: <Intel 82801I (ICH9) USB controller> port 0x1840-0x185f irq 16 at
device 29.0 on pci0
usbus2 on uhci1
ehci1: <Intel 82801I (ICH9) USB 2.0 controller> mem
0xde101400-0xde1017ff irq 16 at device 29.7 on pci0
usbus3: EHCI version 1.0
usbus3 on ehci1
pcib28: <PCI-PCI bridge> at device 30.0 on pci0
pci28: <PCI bus> on pcib28
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
ichsmb0: <Intel 82801I (ICH9) SMBus controller> port 0x1860-0x187f mem
0xde101800-0xde1018ff irq 17 at device 31.3 on pci0
smbus0: <System Management Bus> on ichsmb0
smb0: <SMBus generic I/O> on smbus0
pci0: <dasp> at device 31.6 (no driver attached)
cpu0 on motherboard
coretemp0: <CPU On-Die Thermal Sensors> on cpu0
est0: failed to enable SpeedStep
p4tcc0: <CPU Frequency Thermal Control> on cpu0
cpu1 on motherboard
coretemp1: <CPU On-Die Thermal Sensors> on cpu1
est1: failed to enable SpeedStep
p4tcc1: <CPU Frequency Thermal Control> on cpu1
cpu2 on motherboard
coretemp2: <CPU On-Die Thermal Sensors> on cpu2
est2: failed to enable SpeedStep
p4tcc2: <CPU Frequency Thermal Control> on cpu2
cpu3 on motherboard
coretemp3: <CPU On-Die Thermal Sensors> on cpu3
est3: failed to enable SpeedStep
p4tcc3: <CPU Frequency Thermal Control> on cpu3
ichwd0 on isa0
ichwd0: ICH WDT present but disabled in BIOS or hardware
device_attach: ichwd0 attach returned 6
ipmi0: <IPMI System Interface> on isa0
ipmi0: KCS mode found at io 0xca2 alignment 0x1 on isa
ichwd0 at port 0x1030-0x1037,0x1060-0x107f on isa0
ichwd0: ICH WDT present but disabled in BIOS or hardware
device_attach: ichwd0 attach returned 6
orm0: <ISA Option ROMs> at iomem
0xc0800-0xc17ff,0xc1800-0xc27ff,0xdc000-0xdffff on isa0
atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
atkbd0: [GIANT-LOCKED]
uart0: <16550 or compatible> at port 0x3f8-0x3ff irq 4 flags 0x10 on
isa0
uart0: console (9600,n,8,1)
uart1: <16550 or compatible> at port 0x2f8-0x2ff irq 3 on isa0
atrtc0: <AT realtime clock> at port 0x70 irq 8 on isa0
Event timer "RTC" frequency 32768 Hz quality 0
attimer0: <AT timer> at port 0x40 on isa0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
Timecounters tick every 1.000 msec
iw_cxgb: Chelsio T3 RDMA Driver loaded
random: unblocking device.
usbus0: 12Mbps Full Speed USB v1.0
usbus1: 480Mbps High Speed USB v2.0
ugen0.1: <Intel> at usbus0
uhub0: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0
ugen1.1: <Intel> at usbus1
uhub1: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus1
usbus2: 12Mbps Full Speed USB v1.0
usbus3: 480Mbps High Speed USB v2.0
ugen2.1: <Intel> at usbus2
uhub2: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus2
ugen3.1: <Intel> at usbus3
uhub3: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus3
ipmi0: IPMI device rev. 1, firmware rev. 1.00, version 2.0
ipmi0: Number of channels 0
ipmi0: Attached watchdog
bootpc_init: wired to interface 'em4'
Sending DHCP Discover packet from interface em4 (00:a0:98:30:c2:28)
uhub0: 2 ports with 2 removable, self powered
uhub2: 2 ports with 2 removable, self powered
uhub1: 2 ports with 2 removable, self powered
uhub3: 2 ports with 2 removable, self powered
ugen3.2: <Micron Technology> at usbus3
umass0: <Micron Technology Real SSD eUSB 2GB, class 0/0, rev 2.00/11.10,
addr 2> on usbus3
da0 at umass-sim0 bus 0 scbus0 target 0 lun 0
da0: <MICRON eUSB DISK 1110> Fixed Direct Access SCSI-0 device
da0: Serial Number 0FF0022700196137
da0: 40.000MB/s transfers
da0: 1936MB (3964928 512 byte sectors: 255H 63S/T 246C)
da0: quirks=0x2<NO_6_BYTE>
ugen1.2: <SanDisk> at usbus1
umass1: <SanDisk Extreme, class 0/0, rev 2.10/0.10, addr 2> on usbus1
em4: link state changed to UP
Received DHCP Offer packet on em4 from 192.168.0.1 (accepted) (no root
path) (boot_file)
Received DHCP Offer packet on em4 from 192.168.0.1 (ignored) (no root
path) (boot_file)
da1 at umass-sim1 bus 1 scbus1 target 0 lun 0
da1: <SanDisk Extreme 0001> Removable Direct Access SCSI-6 device
da1: Serial Number AA011208121048257561
da1: 40.000MB/s transfers
da1: 30533MB (62533296 512 byte sectors: 255H 63S/T 3892C)
da1: quirks=0x2<NO_6_BYTE>
Sending DHCP Request packet from interface em4 (00:a0:98:30:c2:28)
Received DHCP Ack packet on em4 from 192.168.0.1 (accepted) (no root
path) (boot_file)
em4 at 192.168.1.1 server 192.168.0.1 boot file /pxe/pxelinux.0
subnet mask 255.255.0.0 router 192.168.0.1 rootfs
192.168.0.1:/usr/home/elars/dst hostname one
Adjusted interface em4
SMP: AP CPU #2 Launched!
SMP: AP CPU #3 Launched!
SMP: AP CPU #1 Launched!
Timecounter "TSC-low" frequency 1500085939 Hz quality 1000
hwpmc: SOFT/16/64/0x67<INT,USR,SYS,REA,WRI> TSC/1/64/0x20<REA>
IAP/2/40/0x3ff<INT,USR,SYS,EDG,THR,REA,WRI,INV,QUA,PRC>
IAF/3/40/0x67<INT,USR,SYS,REA,WRI>
Trying to mount root from nfs: []...
NFS ROOT: 192.168.0.1:/usr/home/elars/dst
Running ipoib_demux (0xffffffff805f5ff0)
mlx4_core0: <mlx4_core> mem 0xdd800000-0xdd8fffff,0xde800000-0xdeffffff
irq 16 at device 0.0 on pci17
<6>mlx4_core: Mellanox ConnectX core driver v1.1 (Dec, 2011)
<6>mlx4_core: Initializing mlx4_core
em0: link state changed to UP
em1: link state changed to UP
em2: link state changed to UP
em3: link state changed to UP
Interface em4 IP-Address 192.168.1.1 Broadcast 192.168.255.255
Entropy harvesting: interrupts ethernet point_to_point swi.
eval: cannot create /etc/hostid: Read-only file system
/etc/rc: WARNING: could not store hostuuid in /etc/hostid.
Writing entropy file:.
Additional TCP/IP options: IPv6 Privacy Addresses.
cxgb0: link state changed to DOWN
mlx4_core0: 64B EQEs/CQEs supported by the device but not enabled
<6>mlx4_en: Mellanox ConnectX HCA Ethernet driver v1.5.2 (July 2010)
mlx4_core1: <mlx4_core> mem 0xdd900000-0xdd9fffff,0xdf000000-0xdf7fffff
irq 17 at device 0.0 on pci18
<6>mlx4_core: Initializing mlx4_core
cxgb1: link state changed to DOWN
em0: link state changed to DOWN
ix0: link state changed to UP
em1: link state changed to DOWN
em2: link state changed to DOWN
em3: link state changed to DOWN
Starting dhclient.
DHCPDISCOVER on em4 to 255.255.255.255 port 67 interval 7
DHCPOFFER from 192.168.0.1
unknown dhcp option value 0xd0
unknown dhcp option value 0xd1
unknown dhcp option value 0xd2
unknown dhcp option value 0xd3
ix1: link state changed to UP
em1: link state changed to UP
DHCPREQUEST on em4 to 255.255.255.255 port 67
DHCPACK from 192.168.0.1
unknown dhcp option value 0xd0
unknown dhcp option value 0xd1
unknown dhcp option value 0xd2
unknown dhcp option value 0xd3
bound to 192.168.1.1 -- renewal in 21600 seconds.
bge0: link state changed to DOWN
em0: link state changed to UP
bge1: link state changed to DOWN
em2: link state changed to UP
em3: link state changed to UP
bge0: link state changed to UP
bge1: link state changed to UP
Starting Network: lo0 cxgb0 cxgb1 ix0 ix1 em0 em1 em2 em3 em4 em5 bge0
bge1.
Starting pflogd:
add net fe80::: gateway ::1
add net ff02::: gateway ::1
add net ::ffff:0.0.0.0: gateway ::1
add net ::0.0.0.0: gateway ::1
Mounting NFS file systems:.
Mounting late file systems:.
Starting rpcbind.
Starting local daemons:.
Performing sanity check on sshd configuration.

Tue Jan 21 09:20:07 CET 2014

FreeBSD/amd64 (one) (ttyu0)

login: mlx4_core1: 64B EQEs/CQEs supported by the device but not enabled
<6>mlx4_ib: Mellanox ConnectX InfiniBand driver v1.0 (April 4, 2008)
<7>ib0: max_srq_sge=31
<7>ib0: max_cm_mtu = 0x10000, num_frags=16
ib0: Attached to mlx4_0 port 1
<7>ib1: max_srq_sge=31
<7>ib1: max_cm_mtu = 0x10000, num_frags=16
ib1: Attached to mlx4_0 port 2
<6>mlx4_ib: Mellanox ConnectX InfiniBand driver v1.0 (April 4, 2008)
<7>ib2: max_srq_sge=31
<7>ib2: max_cm_mtu = 0x10000, num_frags=16
ib2: Attached to mlx4_1 port 1
<7>ib3: max_srq_sge=31
<7>ib3: max_cm_mtu = 0x10000, num_frags=16
ib3: Attached to mlx4_1 port 2


FreeBSD/amd64 (one) (ttyu0)

login:



Received on Tue Jan 21 2014 - 08:05:51 UTC

This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:40:46 UTC