Hello Weijia,
I'm not seeing performance drops on different byte size. Please see below.
[root@mti-mar-s5 ~]# ib_send_bw -s 65535 --report_gbits
************************************
* Waiting for client to connect... *
************************************
---------------------------------------------------------------------------------------
Send BW Test
Dual-port : OFF Device : mlx5_1
Number of qps : 1 Transport type : IB
Connection type : RC Using SRQ : OFF
RX depth : 512
CQ Moderation : 100
Mtu : 4096[B]
Link type : Ethernet
Gid index : 0
Max inline data : 0[B]
rdma_cm QPs : OFF
Data ex. method : Ethernet
---------------------------------------------------------------------------------------
local address: LID 0000 QPN 0x01e5 PSN 0xd8cff4
GID: 00:00:00:00:00:00:00:00:00:00:255:255:12:12:12:05
remote address: LID 0000 QPN 0x01e5 PSN 0xe4ea8a
GID: 00:00:00:00:00:00:00:00:00:00:255:255:12:12:12:06
---------------------------------------------------------------------------------------
#bytes #iterations BW peak[Gb/sec] BW average[Gb/sec] MsgRate[Mpps]
Conflicting CPU frequency values detected: 1200.042000 != 1244.515000. CPU Frequency is not max.
65535 1000 0.00 97.64 0.186239
---------------------------------------------------------------------------------------
[root@mti-mar-s5 ~]# ib_send_bw -s 65536 --report_gbits
************************************
* Waiting for client to connect... *
************************************
---------------------------------------------------------------------------------------
Send BW Test
Dual-port : OFF Device : mlx5_1
Number of qps : 1 Transport type : IB
Connection type : RC Using SRQ : OFF
RX depth : 512
CQ Moderation : 100
Mtu : 4096[B]
Link type : Ethernet
Gid index : 0
Max inline data : 0[B]
rdma_cm QPs : OFF
Data ex. method : Ethernet
---------------------------------------------------------------------------------------
local address: LID 0000 QPN 0x01e6 PSN 0x87adfd
GID: 00:00:00:00:00:00:00:00:00:00:255:255:12:12:12:05
remote address: LID 0000 QPN 0x01e6 PSN 0x8fcb08
GID: 00:00:00:00:00:00:00:00:00:00:255:255:12:12:12:06
---------------------------------------------------------------------------------------
#bytes #iterations BW peak[Gb/sec] BW average[Gb/sec] MsgRate[Mpps]
Conflicting CPU frequency values detected: 1200.132000 != 1268.234000. CPU Frequency is not max.
65536 1000 0.00 97.66 0.186275
---------------------------------------------------------------------------------------
[root@mti-mar-s5 ~]# ib_send_bw -s 65537 --report_gbits
************************************
* Waiting for client to connect... *
************************************
---------------------------------------------------------------------------------------
Send BW Test
Dual-port : OFF Device : mlx5_1
Number of qps : 1 Transport type : IB
Connection type : RC Using SRQ : OFF
RX depth : 512
CQ Moderation : 100
Mtu : 4096[B]
Link type : Ethernet
Gid index : 0
Max inline data : 0[B]
rdma_cm QPs : OFF
Data ex. method : Ethernet
---------------------------------------------------------------------------------------
local address: LID 0000 QPN 0x01e7 PSN 0x4d3748
GID: 00:00:00:00:00:00:00:00:00:00:255:255:12:12:12:05
remote address: LID 0000 QPN 0x01e7 PSN 0x84c3b
GID: 00:00:00:00:00:00:00:00:00:00:255:255:12:12:12:06
---------------------------------------------------------------------------------------
#bytes #iterations BW peak[Gb/sec] BW average[Gb/sec] MsgRate[Mpps]
Conflicting CPU frequency values detected: 1199.953000 != 1225.828000. CPU Frequency is not max.
65537 1000 0.00 97.51 0.185982
---------------------------------------------------------------------------------------
[root@mti-mar-s5 ~]#
Cheers,
~Rage