Hi J. I've spoken to people at Mellanox since I posted my question and they said the FDR cable will work. Thanks for looking into it.
↧
Re: Active Optical Cable for 40GbE cards
↧
I have a MCX354A-FCBT installed on a PCI-3.0 slot of a SM X9DRi-LN4F . Ho do I make sure that FDR infinband data rate full speed on reads and writes.
I have a MCX354A-FCBT installed on a PCI-3.0 slot of a SM X9DRi-LN4F+. How do I make sure that FDR infinband data rate full speed on reads and writes. Thanks.
↧
↧
Re: What does it mean that my MT25204 adapter has an PSID like this: SM_2001000001 (not starting with MT_)?
Thank you yairi. This was very helpful.
↧
Re: SR-IOV on ESXi 5.1
Hi mruepp,
OFED 1.8.1 for ESX Server does not support SRIOV.
Please contact us for roadmap and availability.
↧
Re: I have a MCX354A-FCBT installed on a PCI-3.0 slot of a SM X9DRi-LN4F . Ho do I make sure that FDR infinband data rate full speed on reads and writes.
Hi Irfan,
i am not familiar with the specific model of the server you mentioned but generally speaking this is what you need to get done for gaining the most out of your card:
- install the card on a PCI-e Gen-3 capable PCI slot. choose the one that supports x8 or x16 speeds. Make sure gen-3 is enabled in BIOS (i've seen gen-3 systems that were shipped with factory defaults supporting gen-2)
- load OS (linux or Windows) with InfiniBand driver - shoot for having the latest and greatest of either the OFED community or Mellanox OFED (stick to GA versions)
- make sure your card runs the latest firmware
- There are few protocols that runs on top of IB. some utilize the transport efficiently; some other less. if you want to see wire-speed rates, shoot for using some native IB protocol like RDMA or other protocol that natively speaks with the IB verbs layer - this is a KEY part.since you didn't mention what you application is like, i can't really comment further.
That's it - start with the above and see how far you can go. let me know..
↧
↧
Re: I have a MCX354A-FCBT installed on a PCI-3.0 slot of a SM X9DRi-LN4F . Ho do I make sure that FDR infinband data rate full speed on reads and writes.
If latency is important to you then NUMA affinity really makes a difference. If you run your test (i.e. ib_write_lat) in the same socket where the FDR card is connected, you can achieve latencies under 1 usec. Running in another socket will be ~20% slower. This does not affect the throughput, not in a noticeable way anyways.
IRQ affinity is also very important. BIOS setting too.
There is a very nice, and relatively short, Tuning Guide published by Mellanox which I think is a must: http://www.mellanox.com/related-docs/prod_software/Performance_Tuning_Guide_for_Mellanox_Network_Adapters.pdf
↧
mrg kernel(3.2.23-rt37.56.el6rt.x86_64) update
hi.
red hat kernel is 3.2.23-rt37.56.el6rt.x86_64
mlnx_OFED did not install it.
what should I do?
should we install open source(OFED)?
could you please explain me?
how?
↧
Re: ConnectX-En 10G crash under load :-(
I have tried to install 1.8.1 ofed driver on a freshly installed esxi machine. Esxi shows mt26448 as storage adapter, nothing appears on network adapter.
What should i do now?
Regards.
↧
Re: mrg kernel(3.2.23-rt37.56.el6rt.x86_64) update
Hi,
if this is RH MRG (Real-Time) kernel, unfortunately it is not going to work. Maybe later on this year.
At this point your only option is to use the RH inbox driver (the one that is included with the kernel).
↧
↧
Re: ConnectX-2: mlx4_en: Port: 1, invalid mac burned: 0x0, quiting
Just to add to yairi's note, you can also flip from IB to Eth by setting this module parameter:
options mlx4_core port_type_array="2,2"
or write directly in procfs:
echo eth > /sys/bus/pci/devices/0000\:20\:00.0/mlx4_port2
echo eth > /sys/bus/pci/devices/0000\:20\:00.0/mlx4_port1
↧
Re: ConnectX-2: mlx4_en: Port: 1, invalid mac burned: 0x0, quiting
Thanks for the hint. I just did that on both connected servers and then rmmod mlx4_en mlx4_core, then modprobe mlx4_en.
No mac-related errors in dmesg this time, with these versions:
ConnectX core driver v1.0-mlnx_ofed1.5.3 (November 3, 2011)
ConnectX HCA Ethernet driver v1.5.8.3 (June 2012)
But ifconfig -a still doesn't show any additional adapters.
Firmware is still at 2.8 on both adapters, will try updating them next using mstflint since I can't figure out which flag makes mlxburn accept a simple .bin firmware (that I can get from [1]).
↧
Re: How can I find the firmware version?
I took a stab and ordered them, worst case I'll return them. If I'm correct all I'd have to do is assign static IPs and I should be ok?
↧
Re: ConnectX-2: mlx4_en: Port: 1, invalid mac burned: 0x0, quiting
Does this show eth after you reloaded the modules?
cat /sys/bus/pci/devices/0000\:20\:00.0/mlx4_port1
↧
↧
Re: How can I find the firmware version?
run opensmd (OpenSM Daemon) on one of the machines and then assign static addresses.
↧
Re: How can I find the firmware version?
Could you point me in the direction of any windows docs for this? Opensm is running as a service on the machine .. not familiar with the daemon
↧
Get ready for a new product announcement tomorrow!
If cloud infrastructures are part of your business, we have something for you. Stay tuned
↧
Re: MHGH28-XTC not working
Hmm,
I seem to be hitting a wall again.
The above mentioned flashes worked fine for my two Rev A1 cards but my two rev A2 cards just sit there taking 90% cpu and appear to be doing nothing. There is no text output after I type the command in and hit return.
mft status reports as you would expect (see posts above) but mlxburn and flint both just sit there with a blank line and 90% cpu.
flint -d device_name -q reports that it cannot get semaphores (63)
I have run flint -clear-semaphore -d device_id which seems to complete fine but mlxburn and flint for flashing just sits there doing nothing when run after.
I will try the exact command Todd used for his A2-A3 card but I suspect it is the same as I have been running already.
Update: No luck with the command Todd used.
I also see a lot of errors (IB_Timeouts) in the /vat/log/opensm.log on Linux which is running OpenSM. THis one jumps out at me (see attachment for them all).
Apr 17 21:00:06 008048 [7A680700] 0x01 -> sm_mad_ctrl_send_err_cb: ERR 3120 Timeout while getting attribute 0x15 (PortInfo); Possible mis-set mkey?
Suggestions ?.
↧
↧
Re: MHGH28-XTC not working
@rimblock - As a curiosity question, are you definitely using the A2/A3 firmware? It's a different firmware download than the A1:
http://www.mellanox.com/downloads/firmware/fw-25408-2_9_1000-MHGH28-XTC_A2-A3.bin.zip
Just in case you forgot.
↧
Re: ConnectX-3 CX312A-XCBT and Thunderbolt
@yairi - There's more info being requested here:
permezel wrote:
<snip>
The EchoExpressPro Link Capability register is 333fc41
Bit 22 Reserved = 0
<21> = 1
<20> = 1
<11:10> = 3
Don't know if it enforces any of these to be certain values, in particular <11:10>.
I can find no clear description of the PCI config space settings in any of the Mellanox docs.
I am used to having an explicit (often wrong) enumeration of the PCI config space settings for a device.
If there some other manual I am missing?
<snip>
↧
Re: ConnectX-2: mlx4_en: Port: 1, invalid mac burned: 0x0, quiting
Apparently the port-type gets reset to ib after I rmmod/modprobe mlx4_core:
# echo eth > /sys/bus/pci/devices/0000\:06\:00.0/mlx4_port2
# echo eth > /sys/bus/pci/devices/0000\:06\:00.0/mlx4_port1
# dmesg | tail -n 3
mlx4_en: Mellanox ConnectX HCA Ethernet driver v1.5.8.3 (June 2012)
mlx4_en 0000:06:00.0: Activating port:1
mlx4_en: 0000:06:00.0: Port 1: Port: 1, invalid mac burned: 0x0, quiting
# rmmod mlx4_en mlx4_core
# modprobe mlx4_core
# modprobe mlx4_en
# cat /sys/bus/pci/devices/0000\:06\:00.0/mlx4_port*
ib
ib
↧