Unfortunately answer is no!
But my friend said to me a info some works in advanced.
He test with StarWind vSAN iSER Target and ESXi 6.5 iSER initiator (1.0.0.1) connect to iSER Target.
He test with RoCE v1.0, 1.2x, 2.0.
In near future, I'll test what RoCE version is support with ESXi 6.5 iSER 1.0.0.1 driver.
BR,
Jae-Hoon Choi
regarding the switch side:
it should be
Hello,
Here are the ways to disable FEC on the card:
a.verify with mlxconfig that KEEP_ETH_LINK_UP_P1 value is 0 (for example: mlxconfig -d /dev/mst/mt4117_pciconf0 q)
b.disable RS FEC - "mlxlink -d /dev/mst/mt4117_pciconf0 --fec NF"
c.Toggle link - "mlxlink -d /dev/mst/mt4117_pciconf0 -a TG"
d.verify FEC is disabled - "mlxlink -d /dev/mst/mt4117_pciconf0 --show_fec |grep -i fec"
FEC : No FEC
The MFT (Mellanox firmware tool) can be downloaded from Mellanox fabric website.
Regards,
Viki
Can someone help me to configure my MCX354A-FCBT Mellanox InfiniBand speed at 56Gbps. I have a MCX354A-FCBT Mellanox configured for InfiniBand but the speed remains at 40Gbps (All the components can speed at 56Gbps (card/Switch/Cable)! Thank a lot for your help. Here is my configuration:
Fedora release 24 (Twenty Four)
kernel 4.11.12-100.fc24.x86_64
MCX354A-FCBT
[root@aigle ~]# mlxconfig -d /dev/mst/mt4099_pci_cr0 q
Device #1:
----------
Device type: ConnectX3
PCI device: /dev/mst/mt4099_pci_cr0
Configurations: Next Boot
SRIOV_EN True(1)
NUM_OF_VFS 8
LINK_TYPE_P1 VPI(3)
LINK_TYPE_P2 VPI(3)
LOG_BAR_SIZE 3
BOOT_PKEY_P1 0
BOOT_PKEY_P2 0
BOOT_OPTION_ROM_EN_P1 True(1)
BOOT_VLAN_EN_P1 False(0)
BOOT_RETRY_CNT_P1 0
LEGACY_BOOT_PROTOCOL_P1 PXE(1)
BOOT_VLAN_P1 1
BOOT_OPTION_ROM_EN_P2 True(1)
BOOT_VLAN_EN_P2 False(0)
BOOT_RETRY_CNT_P2 0
LEGACY_BOOT_PROTOCOL_P2 PXE(1)
BOOT_VLAN_P2 1
IP_VER_P1 IPv4(0)
IP_VER_P2 IPv4(0)
[root@aigle ~]# mlxfwmanager --query
Querying Mellanox devices firmware ...
Device #1:
----------
Device Type: ConnectX3
Part Number: MCX354A-FCB_A2-A5
Description: ConnectX-3 VPI adapter card; dual-port QSFP; FDR IB (56Gb/s) and 40GigE; PCIe3.0 x8 8GT/s; RoHS R6
PSID: MT_1090120019
PCI Device Name: /dev/mst/mt4099_pci_cr1
Port1 GUID: f45214030027f751
Port2 GUID: f45214030027f752
Versions: Current Available
FW 2.42.5000 N/A
PXE 3.4.0752 N/A
Status: No matching image found
Device #2:
----------
Device Type: ConnectX3
Part Number: MCX354A-FCB_A2-A5
Description: ConnectX-3 VPI adapter card; dual-port QSFP; FDR IB (56Gb/s) and 40GigE; PCIe3.0 x8 8GT/s; RoHS R6
PSID: MT_1090120019
PCI Device Name: /dev/mst/mt4099_pci_cr0
Port1 GUID: 0002c9030032e311
Port2 GUID: 0002c9030032e312
Versions: Current Available
FW 2.42.5000 N/A
PXE 3.4.0752 N/A
Status: No matching image found
[root@aigle ~]# ibstat
CA 'mlx4_0'
CA type: MT4099
Number of ports: 2
Firmware version: 2.42.5000
Hardware version: 1
Node GUID: 0x0002c9030032e310
System image GUID: 0x0002c9030032e313
Port 1:
State: Active
Physical state: LinkUp
Rate: 40 (FDR10)
Base lid: 6
LMC: 0
SM lid: 1
Capability mask: 0x02514868
Port GUID: 0x0002c9030032e311
Link layer: InfiniBand
Port 2:
State: Active
Physical state: LinkUp
Rate: 40 (FDR10)
Base lid: 7
LMC: 0
SM lid: 1
Capability mask: 0x02514868
Port GUID: 0x0002c9030032e312
Link layer: InfiniBand
CA 'mlx4_1'
CA type: MT4099
Number of ports: 2
Firmware version: 2.42.5000
Hardware version: 1
Node GUID: 0xf45214030027f750
System image GUID: 0xf45214030027f753
Port 1:
State: Active
Physical state: LinkUp
Rate: 40 (FDR10)
Base lid: 8
LMC: 0
SM lid: 1
Capability mask: 0x02514868
Port GUID: 0xf45214030027f751
Link layer: InfiniBand
Port 2:
State: Active
Physical state: LinkUp
Rate: 40 (FDR10)
Base lid: 9
LMC: 0
SM lid: 1
Capability mask: 0x02514868
Port GUID: 0xf45214030027f752
Link layer: InfiniBand
[root@aigle ~]# ibv_devinfo -v
hca_id: mlx4_0
transport: InfiniBand (0)
fw_ver: 2.42.5000
node_guid: 0002:c903:0032:e310
sys_image_guid: 0002:c903:0032:e313
vendor_id: 0x02c9
vendor_part_id: 4099
hw_ver: 0x1
board_id: MT_1090120019
phys_port_cnt: 2
max_mr_size: 0xffffffffffffffff
page_size_cap: 0xfffffe00
max_qp: 393144
max_qp_wr: 16351
device_cap_flags: 0x057e9c76
BAD_PKEY_CNTR
BAD_QKEY_CNTR
AUTO_PATH_MIG
CHANGE_PHY_PORT
UD_AV_PORT_ENFORCE
PORT_ACTIVE_EVENT
SYS_IMAGE_GUID
RC_RNR_NAK_GEN
XRC
Unknown flags: 0x056e8000
device_cap_exp_flags: 0x5000401600000000
EXP_DEVICE_QPG
EXP_UD_RSS
EXP_CROSS_CHANNEL
EXP_MR_ALLOCATE
EXT_ATOMICS
EXP_MASKED_ATOMICS
max_sge: 32
max_sge_rd: 30
max_cq: 65408
max_cqe: 4194303
max_mr: 524032
max_pd: 32764
max_qp_rd_atom: 16
max_ee_rd_atom: 0
max_res_rd_atom: 6290304
max_qp_init_rd_atom: 128
max_ee_init_rd_atom: 0
atomic_cap: ATOMIC_HCA (1)
log atomic arg sizes (mask) 0x8
masked_log_atomic_arg_sizes (mask) 0x8
masked_log_atomic_arg_sizes_network_endianness (mask) 0x0
max fetch and add bit boundary 64
log max atomic inline 3
max_ee: 0
max_rdd: 0
max_mw: 0
max_raw_ipv6_qp: 0
max_raw_ethy_qp: 0
max_mcast_grp: 131072
max_mcast_qp_attach: 244
max_total_mcast_qp_attach: 31981568
max_ah: 2147483647
max_fmr: 0
max_srq: 65472
max_srq_wr: 16383
max_srq_sge: 31
max_pkeys: 128
local_ca_ack_delay: 15
hca_core_clock: 427000
max_klm_list_size: 0
max_send_wqe_inline_klms: 0
max_umr_recursion_depth: 0
max_umr_stride_dimension: 0
general_odp_caps:
max_size: 0x0
rc_odp_caps:
NO SUPPORT
uc_odp_caps:
NO SUPPORT
ud_odp_caps:
NO SUPPORT
dc_odp_caps:
NO SUPPORT
xrc_odp_caps:
NO SUPPORT
raw_eth_odp_caps:
NO SUPPORT
max_dct: 0
max_device_ctx: 1016
Multi-Packet RQ is not supported
rx_pad_end_addr_align: 0
tso_caps:
max_tso: 0
packet_pacing_caps:
qp_rate_limit_min: 0kbps
qp_rate_limit_max: 0kbps
ooo_caps:
ooo_rc_caps = 0x0
ooo_xrc_caps = 0x0
ooo_dc_caps = 0x0
ooo_ud_caps = 0x0
sw_parsing_caps:
supported_qp:
tag matching not supported
Device ports:
port: 1
state: PORT_ACTIVE (4)
max_mtu: 4096 (5)
active_mtu: 4096 (5)
sm_lid: 1
port_lid: 6
port_lmc: 0x00
link_layer: InfiniBand
max_msg_sz: 0x40000000
port_cap_flags: 0x02514868
max_vl_num: 8 (4)
bad_pkey_cntr: 0x0
qkey_viol_cntr: 0x0
sm_sl: 0
pkey_tbl_len: 128
gid_tbl_len: 128
subnet_timeout: 18
init_type_reply: 0
active_width: 4X (2)
active_speed: 10.0 Gbps (8)
phys_state: LINK_UP (5)
GID[ 0]: fe80:0000:0000:0000:0002:c903:0032:e311
port: 2
state: PORT_ACTIVE (4)
max_mtu: 4096 (5)
active_mtu: 4096 (5)
sm_lid: 1
port_lid: 7
port_lmc: 0x00
link_layer: InfiniBand
max_msg_sz: 0x40000000
port_cap_flags: 0x02514868
max_vl_num: 8 (4)
bad_pkey_cntr: 0x0
qkey_viol_cntr: 0x0
sm_sl: 0
pkey_tbl_len: 128
gid_tbl_len: 128
subnet_timeout: 18
init_type_reply: 0
active_width: 4X (2)
active_speed: 10.0 Gbps (8)
phys_state: LINK_UP (5)
GID[ 0]: fe80:0000:0000:0000:0002:c903:0032:e312
Part Info
--------
Type: SX6036
S/N: IL23190198
P/N: 712498-B21
Chassis system GUID: 00:02:C9:03:00:AC:6C:20
Asic FW version: 9.4.3580
LID: 1
Node GUID: 00:02:C9:03:00:AC:6C
Installed MLNX-OS Images
------------------------
Partition 1 - Active Image (partition of next boot)
PPC_M460EX 3.6.4006 2017-07-03 16:17:35 ppc
Partition 2
PPC_M460EX 3.6.3004 2017-02-05 17:31:50 ppc
Port Info
----------
Port number : 1
Port type : IB
IB Subnet : infiniband-default
Port description :
Logical port state : Active
Physical port state : LinkUp
Current line rate : 40.0 Gbps
Supported speeds : sdr, ddr, qdr, fdr10
Speed : fdr10
Supported widths : 1X, 4X
Width : 4X
Max supported MTUs : 4096
MTU : 4096
VL capabilities : VL0 - VL7
Operational VLs : VL0 - VL7
Supported LLR speeds : FDR10, FDR
LLR Status : Active
Identifier : QSFP+
Cable/ Module type : Passive copper, unequalized
Infiniband speeds : SDR , DDR , QDR , FDR
Vendor : Mellanox
Cable length : 2 m
Part number : MC2207130-002
Revision : A3
Serial number : MT1710VS05863
Subnet Manager (SM) Status
--------------------------
SM Status
Local SM running 1 hour 26 minutes 52 seconds
SM Priority 7 State running
Failures 0 Autostart true
Routing Engine Used minhop
SM version OpenSM4.7.0.MLNX20170511.3016205
Hi!
You have SX6036T FDR10 switch.
Your switch's port support up to FDR10 (40Gb, 64/66bit encording).
If you want FDR14(56Gb) speed, you must change your switch to SX6036F or SX6036G.
Best Regard,
Jae-Hoon Choi
in principle, it is possible to achieve optimum bandwidth over "aggregated" Mellanox CX-2 adapters but I'm not sure you can get as much as 40Gb/s using your old platform but then you'll need to ensure the following pre-conditions in your system:
1.In case of Windows platform -
- to line up with the proper OS, driver & fw that supports CX-2 vpi adapter, which is: Windows 8.1, WinOF driver ver. 4.80 & firmware v2.9.120 (Windows 10 is not supported )
- by "teaming" the entire CX-2 adapters on Windows OS (in case teaming is supported in Windows client)
2. the same goes for Linux- you need to line up here as well with the proper OS, driver & fw that supports CX-2 vpi adapter
Can be found in the release-notes of each OS (see link bellow)
http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers
In close, Mellanox CX-2 adapters are already EOF (end-of-life) in terms of sale & Mellanox support. So I suggest you overview the advanced ConnectX-3/4/5 adapters that will provide you the required throughput
per the unsuccessful error print you've presented I can suggest that you use an nvme connect <device> command options that I see is missing there, and that is: "--nr-io-queues"
This option specifies the number of io queues to allocate.
Have you tried this option?
For examples: # nvme connect --transport=rdma --nr-io-queues=36 --trsvcid=4420 --traddr=10.0.1.14 --nqn=test-nvm
Otherwise, you will hit the "default" option which is “num_online_cpus” (Number of controller IO queues that will be established), and this may explains the error you got:
“nvme_fabrics: unknown parameter or missing value 'hostid=a61ecf3f-2925-49a7-9304-cea147f61ae' in ctrl creation request”
read more on that in the article: Add nr_io_queues parameter to connect command: [PATCH v2] nvme-cli/fabrics: Add nr_io_queues parameter to connect command
default:
+ pr_warn("unknown parameter or missing value '%s' in ctrl creation request\n",
+ p);
+ ret = -EINVAL;
+ goto out;
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Hope this helps
Does anyone know how to configure PFC on an esxi 6.5 host for connectx-4 cards?
i followed the following guide to configure linux, however mlnx_qos does not seem to exist for esxi, nor does the python script in the guide:
In UBOOT try running the command:
run flash_self_safe
This will bring the 4036 to the primary kernel where you can recover using "update software"
Unfortunately, iSER 1.0.0.1 isn't beta level.
It's a alpha level quality.
Only 1st port can connect to iSER target.
And Software iSCSI initiator must be add to ESXi 6.5U1 host.
If you don't do it, iSER initiator can't connect to iSER target.
At last, whenever host reboot, iSER initiator disappeared.
You must excute esxcli rdma iser add command to host then rescan HBA adapter everytime.
We are not a beta tester.
We are not a QC employee on Mellanox or VMware,too.
If iSER will be a dominate RDMA storage protocol, It must be add a iSER eanble option to VMware software iSCSI initiator.
Not a funky CLI command.
PFC options and iSER initiator add to ESXi then Host Profile creation work failed everytime since IB iSER beta driver 1.8.3.0.
Various bugs were exist on iSER initiator and PFC option on drivers.
Don't use Mellanox HCA - especially iSER initiator alpha driver 1.0.0.1 - on VMware production environment!
Best Regard.
Jae-Hoon Choi
How did you bind the iscsi adapter for this to work? Is it associated to the same vmnic as the iser hba? did you need to associate a vmkernel adapter with the iscsi initiator at all? If so did you assign an ip address to the iscsi vmkernel? I just dont want to give it a true iscsi path to the target if i do not have to.
can you provide any guidance for esxi pfc setup? i followed this on my linux target but all of the commands seem to not apply to esxi: HowTo Configure PFC on ConnectX-4
Just add a VMware iSCSI adapter.
vmknic must bind iSER initiator, not VMware iSCSI initiator.
IP address also set to vmknic,too.
This test was completed with StarWind vSAN iSER target and RoCE v1 with Global Pause (access port in Arista switch)
This iSER driver 1.0.0.1 for Global Pause(RoCE v1).
Mellanox said this stuff for ESXi 6.5, but QS shows a old ESXi 6.0 C# client picture and ethernet switch must set every port up to Global Pause mode.
This manual is very useless...:(
PFC on ESXi need some configuration with pfcrx, pfctx. But setup the default priority 3 to pfcrx, pfctx to 0x08, you will meet a Host Profile creation error message that general system error. Mellanox never provide a fix since driver 1.8.3.0
BR,
Jae-Hoon Choi
I'm Sorry!
iSER initiator 1.0.0.1 only works with Global Pause.
It doesn't works with PFC on VMware environment.
If you mean that you just using PFC based RDMA network, you must have a Enterprise class ethernet switch that support PFC.
I have 2 of Mellanox SX6036G Gateway switches.
What's your switch model?
BR,
Jae-Hoon Choi
P.S
RDMA must need a switched fabric like FCoE.
If you using direct connection between CX-4, you can't do it!
i do not have a switch, they are working back to back though. there are also a few examples on mellanox's site of direct connect servers as part of their demo. You can configure global pause and pfc in firmware, outside of OS. not sure why this would not work back to back if both adapters are sending pfc/global pause info?
once i enabled iscsi software adapter i was able to connect to my scst target. It is working now. Not really performing any better though. I was able to get better performance in iscsi by configuring 4 vmkernel adapters to my 1 vnic and then configuring round robin policy to 1 iops. It was the same adapter but it seemed to trick esxi into dedicating more hardware resources/scheduler to the adapter. At the moment the iser connection is on par with iscsi before i did this round robin policy. It doesnt look like i can configure round robin policy for iser adapters, but i am still looking into that.
Yes! CX-4 has a advanced unique logic on firmware over CX-3, but that logic must co-works with switch's firmware and configuration.
RoCE RDMA is a kernel bypass protocol.
If your any HCA is down, you meet a kernel dump like ESXi PSOD or Linux kernel dump.
If there isn't switch between host and storage, who control congestion or etc?
This link HowTo Configure PFC on ConnectX-4 that you mentioned previous message include basic system requirements.
BR,
Jae-Hoon Choi
Here is a result from a vm, it has 2 cpus, 8gb and running 2016. It has a vmxnet3 adapter and vmware paravirtual iscsi hard drive.
These results are slightly better than what i was able to achieve with iscsi, so there is some improvement. I understand a switch is recommended but it clearly does work without one. I am monitoring for any packet loss/data issues but as explained before its point to point. The target is 3 nvme pciessds in what amounts to a raid 0 in a zfs pool/zvol btw.
It's a ZFS memory cache ARC range.
You must increase test sample size to 100GB or above.
Best Practice is using iometer with multi-user configurations...:)
BR,
Jae-Hoon Choi
Understood, i am testing the interconnect though, where its coming from is a bit irrelevant. Just saying that iser is performing somewhat better than iscsi so it is worth going through this hassle for those sitting on the sidelines wondering =). I was never able to get above 8800 or so on iscsi. The non sequential are very close, not a huge difference.
I've encountered the same problem with the latest centos7 inbox rpms. Back to mofed it seems.