I think we have every right to ask for clarifications from Mellanox at this stage. ophirmaor?
↧
Re: Which ESXi driver to use for SRP/iSER over IB (not Eth!)?
↧
Re: Which ESXi driver to use for SRP/iSER over IB (not Eth!)?
Some corrections to your list:
* 1.8.2.4 is for 5.x, 1.8.2.5 is for 6.0
* 1.8.3 beta supports both SRP and iSER and can be forcibly installed on 5.x and 6.0
* All 1.8.x.x support X2, X3 and X3 Pro. Also, they're the last to support X2. Not sure about the EN mode, haven't tried it.
* 1.9.x, 2.x and 3.x support only X3 and X3 Pro, none support X2 or older.
* 1.9.x and 3.x support only EN. 1.9.x is the only one supporting iSER.
* 2.x is the latest supporting the IB mode (only for X3 and X3 Pro). It may also support the EN mode, but I haven't tested it.
* Connect-IB, X4 in the IB mode and X5 aren't supported at all - I think this one is particularly insulting, because it means even relatively new cards are left without ESXi support.
I think you're absolutely right with your conclusion to stick with 1.8.2.5 on SRP under ESXi 6.0, this is exactly what I intend to do, even though I have X3 (not Pro) across the board and a managed IB/EN switch. Theoretically, I could use the 1.9.x in the EN mode (still on ESXi 6.0) over iSER, but performance wouldn't be on the same level as SRP and it wouldn't allow me to move to ESXi 6.5 anyway. I don't need any Windows support, my only storage client are ESXi hosts.
As for using X2 as 10Gb NICs, I think this is how they're recognised by the inbox ESXi 6.0 drivers (although not 100% sure). You can give it a shot.
↧
↧
Re: Which ESXi driver to use for SRP/iSER over IB (not Eth!)?
So just summarizing the options:
| Driver Version | Storage Protocol | Adapter Mode | Adapter Family | VMware Ver Supported |
| 1.8.2.5 | IPoIB+SRP | VPI Only | CX-2 | ESXi5.5,ESXi6.0 |
| 1.8.3beta | IPoIB+iSer | VPI Only | CX-2,CX-3 Pro | ESXi5.5,ESXi6.0 |
| 1.9 | iSer | EN Only | CX-3 Pro | ESXi5.5,ESXi6.0 |
| 2 | IPoIB iSCSI | VPI Only | CX-2,CX-3 Pro | ESXi5.5,ESXi6.0 |
| 3 | native iSCSI | EN Only | CX-3 Pro | ESXi5.5,ESXi6.0,ESXi6.5 |
| 4 | native iSCSI | EN Only | CX-4 | ESXi5.5,ESXi6.0,ESXi6.5 |
(see later post with corrected chart)
IPoIB is TCP/IP in software only (no RDMA) on VPI only
No support for any CX-2 solutions
So since I don't have a full complement of CX-3 Pro everywhere and only an un-managed IB switch I would be best to stick with 1.8.2.5 on SRP under ESXi6.0
Also since I need windows storage support I would be best to stick with SRP on 2008R2 since there is no iSer support on windows server. (and no SRP support after 2008R2)
I will stick with ESXi 6.0 currently (probably wouldn't be moving to 6.5 yet anyway) but when I do it looks like I will need to replace the 40G IB switch with an EN switch and fill out my storage network with CX-3 Pro to get iSer support. Also I would need to hopefully find an iSer driver for Server 2012R2 (and/or 2016)
Is there any way with ESXi to use the CX-2 as a 10G ethernet adapter {with appropriate QSFP to SFP adapter} or is there no support at all on ESXi 6.0 (or 6.5).
↧
Re: Which ESXi driver to use for SRP/iSER over IB (not Eth!)?
Ok I think I got all those corrections (might help someone later looking this up)
| Driver Version | Storage Protocol | Adapter Mode | Adapter Family | VMware Ver Supported | Notes |
| 1.8.2.4 | IPoIB+iSCSI, SRP | VPI Only,?EN | CX-2,CX-3,CX-3 Pro | ESXi5.x | |
| 1.8.2.5 | IPoIB+iSCSI, SRP | VPI Only,?EN | CX-2,CX-3,CX-3 Pro | ESXi6.0 | |
| 1.8.3beta | IPoIB+iSCSI, SRP, IPoIB+iSer | VPI Only | CX-2,CX-3,CX-3 Pro | ESXi5.1 | (ESXi5.5 and 6.0 forced) |
| 1.9 | iSCSI, iSer | EN Only | CX-3,CX-3 Pro | ESXi5.5,ESXi6.0 | |
| 2 | IPoIB+iSCSI | VPI,?EN | CX-3,CX-3 Pro | ESXi5.5,ESXi6.0 | |
| 3 | iSCSI | EN Only | CX-3,CX-3 Pro | ESXi5.5,ESXi6.0,ESXi6.5 | |
| 4 | iSCSI | EN Only | CX-4 | ESXi5.5,ESXi6.0,ESXi6.5 |
↧
Re: Peerdirect Raw Ethernet programming
Hi,
I would suggest you to take a look on perftest - git://flatbed.openfabrics.org/~grockah/perftest.git suite. It has support CUDA and raw ethernet.
↧
↧
nfs over rdma
centos 7.3 MLNX_OFED_LINUX-3.4-1.0.0.0
I get this error:
nfs over rdma mount.nfs: mount(2) Input/output error, when hitting this command on the nfs-rdma client:
mount -o rdma,port=20049 ip-Server:/mnt/ /mnt/Client
can anyone give me help?
↧
Re: Windows 2008 R2 BSOD connect x3 WinOF-5.10
Please see the attached screenshot.
Regards,
Jim
↧
Re: Windows 2008 R2 BSOD connect x3 WinOF-5.10
Hi Jim,
Thank you for providing the screenshot.
Can we setup a phone-call regarding this issue? Let me know which time suits you. We can also discuss about moving the case to MyMellanox instead of using the community.
Thanks and regards,
~Martijn
↧
Re: Which ESXi driver to use for SRP/iSER over IB (not Eth!)?
If you use vSphere OFED 1.8.2.4, 1.8.2.5, 1.8.3.0 on ESXi 6.0 with Solaris
COMSTAR SRP, iSER target then you will meet the ESXi PSOD.
ESXi 6.0 support Linux target only.
Jahoon Choi
2017년 1월 30일 (월) 19:03, jasonc <community@mellanox.com>님이 작성:
Mellanox Interconnect Community
<https://community.mellanox.com/?et=watches.email.outcome>
Which ESXi driver to use for SRP/iSER over IB (not Eth!)?
Jason Cecchin
<https://community.mellanox.com/people/jasonc?et=watches.email.outcome>
marked mpogr
<https://community.mellanox.com/people/mpogr?et=watches.email.outcome>'s
reply on Which ESXi driver to use for SRP/iSER over IB (not Eth!)?
<https://community.mellanox.com/thread/3379?et=watches.email.outcome> as
helpful. View the full reply
<https://community.mellanox.com/message/7767?et=watches.email.outcome#comment-7767>
↧
↧
Error messages on nfs load
Im getting error messages in /var/log/messages as soon as nfs used.
casa3 kernel: mlx5_0:mlx5_ib_modify_cq:1060:(pid 2149): modify cq 0x14 failed
were the pid is
| 2149 ? | 00:00:05 ipoib_auto_mode |
Any help would be appreciated!
Daniel F
↧
Re: Storage Spaces Direct Windows Server 2016 (1607) BSOD - Mellanox ConnectX-3 Pro (Dell)
↧
Re: KNEM errors when running OMPI 2.0.1
Hi David,
OMPI can use knem module, however it doesn't care about the compilation. knem is a part of kernel and not a part of OMPI. If any of kernel modules, like knem, cannot be loaded because of wrong symbols the issue should be taken with kernel module developers.
At the same time, you might try to recompile the modules for you kernel and see if it help. This link might be a good start point Command to rebuild all DKMS modules for all installed kernels? - Ask Ubuntu
↧
Re: Mgmt_class for MAD packet in MLX4 driver
Hello Rama,
What driver version are you currently running?
How are you observing the Mgmt_class with rping?
What do you mean by "data offset as per Subn class"?
Is there a current issue?
What are you trying to accomplish?
Sophie.
↧
↧
mlx5 with inbox driver 100G is not detecting
Hi,
I am trying to up 100G Ethernet driver. I am using inbox driver, I am not using MLNX OFED stack.
I am inserting drivers manually in the sequence of ib_cm...mlx5_core,mlx5_ib
mkdir /sys/kernel/config/rdma_cm/mlx5_0
echo "RoCE v2" > /sys/kernel/config/rdma_cm/$mlx5_0/ports/1/default_roce_mode
But with ethtool command is not showing 100G, Am I missing some thing?
root@xhdipsspdk1 ~]# ethtool enp5s0
Settings for enp5s0:
Supported ports: [ FIBRE ]
Supported link modes: 1000baseT/Full
1000baseKX/Full
40000baseKR4/Full
40000baseCR4/Full
40000baseSR4/Full
40000baseLR4/Full
Supported pause frame use: Symmetric Receive-only
Supports auto-negotiation: No
Advertised link modes: 1000baseT/Full
1000baseKX/Full
40000baseKR4/Full
40000baseCR4/Full
40000baseSR4/Full
40000baseLR4/Full
Advertised pause frame use: No
Advertised auto-negotiation: No
Speed: Unknown!
Duplex: Unknown! (255)
Port: FIBRE
PHYAD: 0
Transceiver: internal
Auto-negotiation: off
Supports Wake-on: d
Wake-on: d
Supports Wake-on: d
Wake-on: d
Link detected: no
↧
packet drops in the Rx port
Hello,
I am using ConnectX-3 with driver version 3.4-1 (ethtool --driver command). It has 16 RX rings, and I set the ring size to 8192 (ethtool -g shows 8192 RX buffer). I see the interface sometimes drops packets.
# ethtool -S eth1 | grep drop
rx_dropped: 152
vport_rx_dropped: 0
rx0_dropped: 0
....
rx15_dropped:0
# ethtool -S eth1 | grep over
rx_over_errors: 152
Interrupts from each Rx rings are handled by different CPU core. I monitored Rx packet count, Rx drop count, and interrupt counts. When the drop happened, the interface received about 6000 frames, and with drop count increased by 30. The interrupts were well distributed to each rings. It did not receive any frame for 300 ms before this happened, and the system wasn't busy at all. I have some questions.
1. How the interface drops frames when it received frames much less than the Rx ring buffer size?
2. What's the implication that rx_dropped count increased, but the drop counts of each ring are all zero? Does this mean that it dropped the frame before it reaches the RSS?
3. Does rx_over_errors counter mean that the frame was dropped because it couldn't find the Rx WQE? Is it looking for resources other than Rx WQEs?
I'd appreciate any comments.
Thanks
↧
Imaging an IB Switch with Custom Firmware
I have a SX6018 purchased off of eBay which I want to image via a custom firmware build. Can anyone walk me through some of the steps I need to take to get this thing imaged? Can anyone point me in the right direction to acquire a copy of MLNX-OS?
This switch is the last ingredient to my homelab, but I'm broke so I opted for the cheap option that involves more work.
Best Regards!
↧
Re: Please help
The I2C port only switches are unmanaged. Sorry.
↧
↧
Re: Which ESXi driver to use for SRP/iSER over IB (not Eth!)?
I would like to clarify the ESXi over InfiniBand (IB) support topic and sorry for being late to the thread.
IB Para-virtualization (PV) and SR-IOV are supported only in ESXi versions that support the VMKLinux driver, which means in version ESXi 6.0 or older. In those versions, the IPoIB standard protocol has been implemented. In ESXi 6.5 (that
supports only native driver), Mellanox plans to add IB over SR-IOV support in June’17.
Re. ESXi’s Storage protocols:
- SRP runs only over IB and the latest driver that included new features was 1.8.2.3 over ESXi 5.5 using ConnectX-2, ConnectX-3 & ConnectX-3 Pro. Since then, the SRP driver is in a maintenance mode (means, Mellanox will only fix issues).
- iSER support comes only over RoCE and ESXi 6.5 includes inbox Mellanox’s TCP/IP & RoCE drivers over all speeds, 10, 25, 40, 50 & 100 Gb/s and run over ConnectX-4 Lx and ConnectX-4. ConnectX-5 support will be added later this year.
↧
Re: Which ESXi driver to use for SRP/iSER over IB (not Eth!)?
Okay!
I want to hear it.
ConnectX-3 family will be drop to support RoCE iSER on ESXi 6.5 and above...:(
My friend also test with MCX354A-FCBT on ESXi 6.5 via esxcli rdma iser add command.
But failed to create iSER initiator.
↧
Re: Which ESXi driver to use for SRP/iSER over IB (not Eth!)?
Thanks for the reply.
So just clarifying a few things:
- So is "IB Para-virtualization (PV)" is actually "Paravirtualization of IP over Infiniband" otherwise known as (IPoIB). This is the tool which allows us to run IP based applications on an IB fabric.
- The SR-IOV features allow a physical NIC to be shared in a virtual environment.
So if you look at my previous table/chart there was support up to ESXi6.0 for most protocols and features because these used the VMKLinux device driver model.
For those who don't know VMKLinux was a carry over from the old ESX (which required Linux) days to allow Linux device drivers to be essentially still used with ESXi (with a few mods) - even through in ESXi Linux doesn't technically exist. In ESXi5.5 VMware released a new "Native Device Driver Model" where the drivers are directly interfacing with the VMKernel now rather than going through the VMKLinux shim compatibility layer. (VMware released the new device driver model so drivers can be more efficient, flexible and have better performance. In addition there is more debugging and troubleshooting features along with support for new capabilities such as hot plugging - see more info here https://blogs.vmware.com/tap/2014/02/vmware-native-driver-architecture-enables-partners-deliver-simplicity-robustness-performance.html)
So you mentioned that SRP protocol support has been retired and won't be carried forward with the new driver model so I am assuming we won't be able to use it on ESXi v6.5 and later versions of ESXi.
Also stated was that ESXi v6.5 includes drivers for CX-4 inbox (and CX-5 drivers are coming later this year) and that these drivers support ROCE and iSer when adapters are run in ethernet mode on an ethernet switch only.
Question still remaining:
- Because of the new native drivers included inbox for ESXi v6.5 the older VMKLinux drivers will not work any longer. Can we disable or remove these drivers and continue to use the older VMKLinux drivers similar to what is described here https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2147565?
- Will there be any support for CX-3 and CX-3Pro adapters under the new Native device driver model for ESXi v6.5 and later? (and therefore support for iSer with CX-3 and CX-3Pro like you provide for CX-4 now (and for CX-5 in the future)?
- Will there be any support for IPoIB on the new Native device driver model or is this not being carried forward either?
Because the way it looks at the moment is if we want to run RDMA accelerated storage on ESXi v6.5 we will need to purchase CX-4 or later adapters (currently only CX-4) and run them on an ethernet switch, otherwise we will be stuck at ESXi v6 due to the lack of CX3 drivers under the new device driver model and inability to use the old ones.
↧