Hi,
Thanks for respons and help.
First of all, Supermicro's mobo supports SR-IOV, VT-D. In this version of motherboard nad BIOS, SR-IOV is turn on all the time.
I forgot to mention when phisical device is passthrough everything works as expected in VM.
On this issue, I'm working on 2 phisical nodes S1 and G1, I have OpenSM on G1
OpenSM (sminfo BUILD VERSION: 1.5.8.MLNX_20110906 Build date: Jun 26 2012 21:31:16))
S1 has virtual system CentOS64
I've started working on infiniband technology about 1 year ago and so far almost everything I needed was out of the box
therefore I can be a little bit clumsy
[root@G1]# sminfo
sminfo: sm lid 2 sm guid 0x8f104039814b9, activity count 3626 priority 0 state 3 SMINFO_MASTER
[root@S1 ~]# sminfo
sminfo: sm lid 2 sm guid 0x8f104039814b9, activity count 3707 priority 0 state 3 SMINFO_MASTER
[root@S1 ~]# sminfo -G 0x8f104039814b9
sminfo: sm lid 2 sm guid 0x8f104039814b9, activity count 3718 priority 0 state 3 SMINFO_MASTER
[root@CentOS64 ~]# sminfo -G 0x8f104039814b9
ibwarn: [3178] ib_path_query_via: sa call path_query failed
sminfo: iberror: failed: can't resolve destination port 0x8f104039814b9
[root@CentOS64 ~]# ibstat
CA 'mlx4_0'
CA type: MT4100
Number of ports: 2
Firmware version: 2.11.500
Hardware version: 0
Node GUID: 0x001405008eaa0a36
System image GUID: 0x0002c90300a28fb3
Port 1:
State: Active
Physical state: LinkUp
Rate: 20
Base lid: 1
LMC: 0
SM lid: 2
Capability mask: 0x02514868
Port GUID: 0x0014050000000002
Link layer: InfiniBand
Port 2:
State: Down
Physical state: Disabled
Rate: 10
Base lid: 4
LMC: 0
SM lid: 1
Capability mask: 0x02514868
Port GUID: 0x0014050000000081
Link layer: InfiniBand
2 devices from: options mlx4_core num_vfs=5 port_type_array=1,1 probe_vf=1
[root@S1 ~]# ibstat
CA 'mlx4_0'
CA type: MT4099
Number of ports: 2
Firmware version: 2.11.500
Hardware version: 0
Node GUID: 0x0002c90300a28fb0
System image GUID: 0x0002c90300a28fb3
Port 1:
State: Active
Physical state: LinkUp
Rate: 20
Base lid: 1
LMC: 0
SM lid: 2
Capability mask: 0x02514868
Port GUID: 0x0002c90300a28fb1
Link layer: InfiniBand
Port 2:
State: Down
Physical state: Disabled
Rate: 10
Base lid: 4
LMC: 0
SM lid: 1
Capability mask: 0x02514868
Port GUID: 0x0002c90300a28fb2
Link layer: InfiniBand
CA 'mlx4_1'
CA type: MT4100
Number of ports: 2
Firmware version: 2.11.500
Hardware version: 0
Node GUID: 0x00140500f8bf4e16
System image GUID: 0x0002c90300a28fb3
Port 1:
State: Active
Physical state: LinkUp
Rate: 20
Base lid: 1
LMC: 0
SM lid: 2
Capability mask: 0x02514868
Port GUID: 0x0014050000000001
Link layer: InfiniBand
Port 2:
State: Down
Physical state: Disabled
Rate: 10
Base lid: 4
LMC: 0
SM lid: 1
Capability mask: 0x02514868
Port GUID: 0x0014050000000080
Link layer: InfiniBand
[root@G1 ]# ibstat
CA 'mthca0'
CA type: MT25208 (MT23108 compat mode)
Number of ports: 2
Firmware version: 4.7.600
Hardware version: a0
Node GUID: 0x0008f104039814b8
System image GUID: 0x0008f104039814bb
Port 1:
State: Active
Physical state: LinkUp
Rate: 20
Base lid: 2
LMC: 0
SM lid: 2
Capability mask: 0x02510a6a
Port GUID: 0x0008f104039814b9
Link layer: InfiniBand
Port 2:
State: Down
Physical state: Polling
Rate: 10
Base lid: 0
LMC: 0
SM lid: 0
Capability mask: 0x02510a68
Port GUID: 0x0008f104039814ba
Link layer: InfiniBand