Yes, that's normal. You need to pass that parameter to modprobe:
# modprobe mlx4_core port_type_array="2,2"
Otherwise it defaults to "ib"
You can also set that options in /etc/modprobe.d/*.conf (or something similar for CentOS) so you don't have to specify it every time you use modprobe.
But you still get that "invalid mac burned" error, which might be related to firmware.
Just updated firmware to 2.9.1000. 'mstflint' warned that PSID's didn't match (originally I had HP_0160000009, it got burned to MT_0D70110009). Same dmesg error about 'invalid mac burned' with this firmware and mlx4_en 1.5.8.3.
I can backup the HP_0160000009 firmware from another board and re-burn it if needed.
I did pass that port_type_array option to modprobe when testing.
I looked at the source code, and as odd as it sounds it looks like your adapter does not have a MAC address (or it's set to 0x0)
ofa_kernel-1.5.3/drivers/net/mlx4/mlx4_en.h:
#define ILLEGAL_MAC(addr)(addr == 0xffffffffffffULL || addr == 0x0)
ofa_kernel-1.5.3/drivers/net/mlx4/en_netdev.c
priv->mac = mdev->dev->caps.def_mac[priv->port];
if (ILLEGAL_MAC(priv->mac)) {
mlx4_err(mdev, "Port: %d, invalid mac burned: 0x%llx, quiting\n",
priv->port, priv->mac);
The serial number reported by lspci should include the mac address, I am curious if that is zero too.
# lspci -vvv -s 06:00.0 | grep Serial
Capabilities: [148] Device Serial Number 00-02-c9-03-00-12-83-44
Windows Server 2012 File Server Performance with EchoStreams FlacheSAN2 and Mellanox ConnectX-3
This article presents a summary of the performance of the Windows Server 2012 File Server’s Remote Direct Memory Access capabilities over SMB 3.0.
Shared with Jive Anywhere
We here at Mellanox receive many questions regarding IPoIB.
What can we answer for you?
Is this also true for the new ConnectX-3 Pro?
According to this http://www.mellanox.com/related-docs/prod_adapter_cards/PB_ConnectX-3_Pro_Card_EN.pdf
ConnectX-3 Pro supports SR-IOV and VMWare, but it doesn't mention which OFED version.
Try updating the FW of the card to the latest. i know that there should be an option in the newer Firmware to create the MAC our of the card's number when the FW starts.
give it a try..
Actually, it's pretty easy to set/change the MAC (page 29 on http://www.mellanox.com/pdf/MFT/MFT_user_manual.pdf)
# mst start
Query the mac:
# flint -d /dev/mst/mt25418_pci_cr0 -qq q
Change the mac:
# flint -d /dev/mst/mt25418_pci_cr0 -mac 02c90abcdef0 sg
Change the GUID:
# flint -d /dev/mst/mt25418_pci_cr0 -guid 0002c9000abcdef0 sg
# flint -d /dev/mst/mt25418_pci_cr0 -qq q
The current setup seems to be working and is this...
Port 1 - Solaris 11.1 - ConnectX-2 (SAN)
Port 2 - ESXi 5.1: ConnectX A2 (unflashed)
Port 3 - ESXi 5.1: none
Port 4 - CentOS 6.4 (OpenSM): ConnectX A1 (flashed)
Port 5 - ESXi 5.1: ConnectX A2 (unflashed)
So, the two core ESXi 5.1 hosts have ver A2, the OpenSM box has a ver A1 and the san has a X-2.
I had to remove the Mellanox ESXi vib and then reinstall it as it did not 'see' the A2 card. After the reinstall the card popped up and the targets were available.
It also seems that ver A1 will not work with VT-d (passthrough). After assigning the card for passthrough on the ESXi host, rebooting and then adding the card to the Windows 2012 Ess VM as a PCI devce, the VM will not start. It produces a caught error (at least not a PSOD). I have not yet tried with the A2 version.
Do you know if the Performance Tuning Script can be run from the command line, such as on the Core or Hyper-V versions of Windows. These do not have the GUI that allows you to open Device Manager to access Advanced Settings and such
Check section 5.1.4.1
http://www.mellanox.com/related-docs/prod_software/MLNX_VPI_WinOF_User_Manual_v4.2.pdf
UD is becoming more performance oriented than ever before, and can cross our IB to Ethernet gateways. CM cannot cross gateways which is a major issue for compatibility.
Linux still has both CM and UD modes, but no other OS that I can see has it (VMware, Windows, Xen, etc).
That being said, RSS, TSS, LSO, LRO should help alleviate UD issues from the past, so hopefully UD will be sufficient for you.
Thanks Jeff! Just what I was looking for.
Sounds good (especially the ubuntu part :-)
Do you know if the beta (and/or GA) supports ConnectX-2 and works with BX5020 and IS5030?
When I asked support, early march, to participate in the beta program they said the beta only supported ConnectX-3 and was without support for BX5020. Hopefully that has changed by now.
OK, I finally received my switch box. I connected everything and nothing happened. I then did as justinclift suggested below. I rebooted and everything seems to be lit up now. Thank you very much justinclift. I am not sure how to configure to ensure that the servers are using or communicating or how to test. I did find this link but it didn't work for me
How to install support for Mellanox Infiniband hardware on RHEL6 - Red Hat Customer Portal
justinclift
"
Yeah, that line is a good sign. It's showing the hardware is being found.
When the switch arrives and you have things hooked up, the easy way to try things is probably this:
$ sudo yum groupinstall "Infiniband Support"
In theory that should install working drivers.
Though, as mentioned by the guys above, Mellanox support might not really like that.
It's pretty easy to remove those packages afterwards though if they want you to try a different approach:
$ sudo yum groupremove "Infiniband Support"
Just saying. "
Cool.
The commands you'll probably find most useful right away are:
The config files you'll probably find most interesting are in the /etc/rdma/ directory.
Out of curiosity, are you wanting to setup IP addressing for the adapters? It's pretty easy to do.
Justin, I think I don't have something configured properly. Here's what I get:
[root@think172 ~]# ibv_devinfo
No IB devices found
[root@think172 ~]# ibaddr
ibwarn: [20819] mad_rpc_open_port: can't open UMAD port ((null):0)
ibaddr: iberror: failed: Failed to open '(null)' port '0'
[root@think172 ~]# ibdiagnet
Loading IBDIAGNET from: /usr/lib64/ibdiagnet1.5.7
-W- Topology file is not specified.
Reports regarding cluster links will use direct routes.
Loading IBDM from: /usr/lib64/ibdm1.5.7
-E- IBIS: No HCA was found on local machine.
Exiting.
[root@think172 ~]# ibnodes
src/query_smp.c:227; can't open UMAD port ((null):0)
/usr/sbin/ibnetdiscover: iberror: failed: discover failed
src/query_smp.c:227; can't open UMAD port ((null):0)
/usr/sbin/ibnetdiscover: iberror: failed: discover failed
[root@think172 ~]# iping
bash: iping: command not found
[root@think172 ~]#
I appreciate your help as I am still new to this. Regarding your question for IP addressing, I'm not sure, but I do know that my intent is to setup this machine for finite element analysis using Ansys RSM (remote solve manager) to do this. I can choose to create a cluster from these machines, or leave them as individual machines. Any suggestions?
Thanks, David.
Hmmm, it kind of sounds like you're just wanting to run the adapters in native 10 GbE mode instead of in IB mode.
That's super simple to do if you're using the Infiniband stuff that comes with CentOS. (the "Infiniband Support" yum group.) It's just a setting you change in one of the /etc/rdma/ conf files.
Oops, I gave you bad info there.
Didn't notice/remember that you're using EN cards. i.e. Ethernet only ones
I don't think the Infiniband specific commands work with them.
Guessing that your adapters should probably show up as ethernet cards instead. eg: eth0 eth1 eth2 or similar.
If that's the case, normal ethernet configuration stuff should work fine.
I have no clue at all about Ansys RSM so no suggestions there.
Thanks, the flint -mac hints were spot on. I can now see the two eth* devices in ifconfig, but when doing:
machine1$ ifconfig eth0 192.192.168.1.1
machine2$ ifconfig eth0 192.192.168.1.2
this comes up in dmesg:
mlx4_en 0000:06:00.0: Activating port:1
mlx4_en: eth0: Using 16 TX rings
mlx4_en: eth0: Using 16 RX rings
mlx4_en: eth0: Initializing port
ADDRCONF(NETDEV_UP): eth0: link is not ready
And ping doesn't work between the machines.
When setting port types to ib and using the ib0 devices, ping works, netperf tests work, etc (I started an opensm service on one machine). I've tried "ifconfig eth0 down" then back up on the machines with no success. I checked with ibdev2netdev that I was using the right eth device (ib0 <==> eth0).
I did "yum groupinstall Infiniband\ Support" and then configured things in /etc/rdma but got the same result ("link is not ready").
Is there anything I missed ? The MLNX_OFED manual doesn't list anything besides ifconfig under "4.1.9 A detailed example".
