Hi Sophie,
1. OS - linux gentoo, i have 'ofed" in the repository (
sys-infiniband/ofed-3.12_rc1:3.12 USE="dapl diags ibacm mstflint opensm perftest qperf -rds -srp" OFED_DRIVERS="mlx4 -cxgb3 -cxgb4 (-ehca) -ipath -mlx5 -mthca -nes -ocrdma -psm" 0 KiB
2. Bios settings (C States - off, Hyper - Threading - enabled)
OS tuning
#!/bin/sh
for cf in /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor;
do
echo performance > $cf;
done
echo 1000000000 > /proc/sys/kernel/shmmax
echo 400 > /proc/sys/vm/nr_hugepages
sysctl -w net.ipv4.tcp_timestamps=0
sysctl -w net.ipv4.tcp_sack=1
sysctl -w net.core.netdev_max_backlog=250000
sysctl -w net.core.rmem_max=4194304
sysctl -w net.core.wmem_max=4194304
sysctl -w net.core.rmem_default=4194304
sysctl -w net.core.wmem_default=4194304
sysctl -w net.core.optmem_max=4194304
sysctl -w net.ipv4.tcp_rmem="4096 87380 4194304"
sysctl -w net.ipv4.tcp_wmem="4096 65536 4194304"
sysctl -w net.ipv4.tcp_low_latency=1
NUMA - not support my CPU
3. All adaptersinstalledinPCIE GEN2x8
4. test speed
RDMA_Write BW Test
Dual-port : OFF Device : mlx4_0
Number of qps : 1 Transport type : IB
Connection type : RC Using SRQ : OFF
CQ Moderation : 100
Mtu : 4096[B]
Link type : IB
Max inline data : 0[B]
rdma_cm QPs : OFF
Data ex. method : Ethernet
---------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------
#bytes #iterations BW peak[Gb/sec] BW average[Gb/sec] MsgRate[Mpps]
8388608 5000 13.72 13.72 0.000204
---------------------------------------------------------------------------------------
qperf 10.0.0.1 rc_bw
rc_bw:
bw = 1.71 GB/sec
It was better but there is no40gb
ps. Version ofed -
OFED_VER="3.12"
OFED_RC="1"
OFED_RC_VER="1"
OFED_SUFFIX="1.g8336567"