Hello David -
I hope all is well...
Please try the following :
# mlnx_qos -i eth2 -p 0,1,2,3,4,5,6,7
-p --prio_tc
This parameter is used to map priorities to Egress Traffic Class (ETC).
Note: By default, priority 0 is mapped to tc1, and priority 1 is mapped to tc0. All other priorities are mapped to the same TC.
Please let me know you results...
Thanks - steve
Hi Steve,
the default output is shown above, except the PFC setting from your previous post.
DCBX mode: OS controlled
Priority trust state: pcp
Cable len: 7
PFC configuration:
priority 0 1 2 3 4 5 6 7
enabled 0 0 0 1 0 0 0 0
tc: 0 ratelimit: unlimited, tsa: vendor
priority: 0
priority: 1
priority: 2
priority: 3
priority: 4
priority: 5
priority: 6
priority: 7
And yes, when I try to set prio values to ETCs I get the errors.
The output with -p or with --prio_tc is listed above
sudo mlnx_qos -i enp6s0f1 -p 0,1,2,3,4,5,6,7
Netlink error: Bad value. see dmesg.
Thank you, David
Hi David,
Could you shed the light on what you are trying to achieve? Why you are configuring SR-IOV? Are you going to use VLAN? Are you planning to run VM on it? and then RDMA traffic over it? Understanding whole picture of the final result will be extremely helpful.
Hi,
yes I'm going to use VLAN, we have a smaller network of containers and we assign SR-IOV VFs to each of them. At the moment we are using 2 servers connected back-to-back. In the containers we are running bandwidth hungry and latency sensitive applications. By setting QoS rules we would like to reduce the message passing latencies of the latency sensitive traffic. Latency values are increasing due to the presence of the bandwidth hungry traffic. We need to use VLAN because of the value of the PCP header, which differentiates the types of traffic. However running 8 containers per server is not a real world scenario I think.
Note: There was a previous question which seemed to be solved. That was about how we cannot assign priority values to TCs of SR-IOV VFs -- furthermore VFs are completely not configurable by the mlnx_qos tool -- but according to our measurements the priority settings of the PF are inherited by the VFs, so it works out.
Thanks,
David
It seems that the RedHat has changed the naming conventions of some RPM packages. In the file "centos-release-7-5.*aarch64" rpm packages, the letter "a" is used instead of the full name "alternate".
So in the script, the checking will fail and refuse to install the package.
# } elsif ($dist_rpm =~ /redhat-release-.*-7.5.*el7a|sl-release-7.5.*el7a|centos-release-7-5.*el7.centos.altarch/) {
} elsif ($dist_rpm =~ /redhat-release-.*-7.5.*el7a|sl-release-7.5.*el7a|centos-release-7-5.*el7.centos.a*/) {
I tried this WA. Would you please give some official solution for this?
Thanks
Hello Romain -
I hope all is well...
Do you have an email address so I can send you a link to the file?
thanks - steve
Thank you. I sent you a direct message with my e-mail.
BTW, naive question - any reason the link can't simply be made public?
(my guess would be, this is somehow a prerelease HCA and there was never any official release for it?)
Cordially & thanks for the great support,
Romain
Hello Community,
I have 2 Mellanox CX516A in same x86 host. I successfully use a Finisar QSFP28 100G LR module plugged into each NIC and back-to-back cabled. Means NIC 1 port 0 goes to NIC 2 port 0, this is working fine.
I tried to place both QSFP28 modules into the same NIC and use back-to-back cabling. After I plugged in the second QSFP28 into port 1 I got the following error message:
[419335.854822] mlx5_core 0000:88:00.1: Port module event[error]: module 1, Cable error, Power budget exceeded
also the orange led is flashing.
My host running Ubuntu 18.04 with "MLNX_OFED_LINUX-4.3-3.0.2.1-ubuntu18.04-x86_64" driver version.
Do you have any hints how to bringup both QSFP28 in the same NIC ?
Hello Romain -
Service for this Hardware and FW has ended. So it will not be hosted on our site.
Many thanks for your support & patience...
Steve
Greetings,
We have some nodes (Dell R415s running RHEL 6.8) with Connect-IB cards in a PCI 2.0 x16 slot (the only one available) and can't seem to get more than 45 Gbit/s using ib_send_bw. I have two of the nodes connected directly using a new FDR cable and with the SM running on one of the nodes. I have updated the BIOS, the OFED and the HCA firmware on both nodes. Still I can't seem to get the full FDR bandwidth. The connect-ib product page (http://www.mellanox.com/page/products_dyn?product_family=142&mtag=connect_ib ) states the following:
"Connect-IB also enables PCI Express 2.0 x16 systems to take full advantage of FDR, delivering at least twice the bandwidth of existing PCIe 2.0 solutions."
Since the PCIe 2.0 x16 is able to support 64 Gbit/s in one direction, shouldn't I be able to achieve full FDR (~54ish Gbit/s) as the product page implies. Or, am I wrong and there is some extra overhead that reduces the bandwidth for PCIe 2.0 x16 vs. PCIe 3.0 x16?
I have gone through the Performance Tuning for Mellanox Adapters guide and there isn't much more that I can try based on this. The latest BIOS has nowhere near the number of setting that are suggested to be tweaked in the guide. I have also tried mlnx_tune and get one warning:
----------------------------------------------------------
Connect-IB Device Status on PCI 01:00.0
FW version 10.16.1200
OK: PCI Width x16
Warning: PCI Speed 5GT/s >>> PCI width status is below PCI capabilities. Check PCI configuration in BIOS. <--------------
PCI Max Payload Size 128
PCI Max Read Request 512
Local CPUs list [0, 1, 2, 3, 4, 5]
----------------------------------------------------------
But this is probably correct since I am using a PCIe 2.0 x16 slot (PCIe 2.0 can only do 5 GT/s), right?
Here is the output of ibv_devinfo:
-------------------------------------------
hca_id: mlx5_0
transport: InfiniBand (0)
fw_ver: 10.16.1200
node_guid: f452:1403:002e:eb40
sys_image_guid: f452:1403:002e:eb40
vendor_id: 0x02c9
vendor_part_id: 4113
hw_ver: 0x0
board_id: MT_1220110019
phys_port_cnt: 1
Device ports:
port: 1
state: PORT_ACTIVE (4)
max_mtu: 4096 (5)
active_mtu: 4096 (5)
sm_lid: 1
port_lid: 2
port_lmc: 0x00
link_layer: InfiniBand
-------------------------------------------
and iblinkinfo:
-------------------------------------------
CA: A HCA-1:
0xf4521403002ee9f0 1 1[ ] ==( 4X 14.0625 Gbps Active/ LinkUp)==> 2 1[ ] "B" ( )
CA: B HCA-1:
0xf4521403002eeb40 2 1[ ] ==( 4X 14.0625 Gbps Active/ LinkUp)==> 1 1[ ] "A" ( )
-------------------------------------------
Can anyone tell me if this is the best I can expect or is there something else I can change to achieve FDR bandwidth with these HCAs?
Thanks in advance!
Eric
Hi Edward,
See the link below to the Mellanox Adapters Programmer’s Reference Manual (PRM)
Supporting ConnectX®-4 and ConnectX®-4 Lx as they are different adapter as from the ConnectX-3.
http://www.mellanox.com/related-docs/user_manuals/Ethernet_Adapters_Programming_Manual.pdf
Sophie.
Hello gents
Pardon this sheep, this is my first experience with any InfiniBand technologies at all.
I've gotten my hands on a SX6012 through some distant work connections that nobody seems to know what to do with it so I figured let's get cracking.
Upon power up, it throws the message "System is initializing! This may take a few minutes" , "Modules are being configured", then it throws "Internal error, code 1006 (see logs for details) A fatal internal error occurred". Then it kicked me out.
I looked in user manual, there's no mention of code 1006, searched this forum, nada. Almighty Google didn't have much either. How can I get that logs for further detail? And what does this code 1006 mean?
Your assistance is much appreciated.
Thank you.
Hi,
Can you show me your output of
rpm -qf /etc/issue on your Centos 7.4 and 7.5 to see the difference
dist_rpm use this output to check after the lines you refer to.
Thanks
Marc
Hi,
How did you set your irq affinity ?
Did you try to use set_irq_affinity_bynode.sh script ?
Try again and let me know
Marc
Hi Marc,
rpm -qf /etc/issue
centos-release-7-5.1804.el7.centos.a.aarch64
In the CentOS 7.5 release
In the last release
Index of /altarch/7.4.1708/updates/aarch64/Packages
You can see that the the naming conventions are a little different of altarch aarch64. I have no idea of this package's name on the Power. Many thanks
Hi Simon,
question: do you want to use Storage Spaces Direct in Windows Server 2016 with it? That is at least my problem.
Cheers Carsten Rachfahl
Microsoft Clud & Datacenter Managment MVP
Hi,
I have ConnectX-2 card in my ESXi 6.5 server
Added this card to FreeNAS (VM) and OpenMediaVault (VM) as VMXNET3 adaptor
When i test the speed between the FreeNAS/OMV (VM) and Windows (PC) i get only 200-300 MBytes/sec
When i test the speed between the FreeNAS (VM) and OMV (VM) i get 4000-5000 MBytes/sec - on the same vSwitch
When i test the speed between the Windows (PC) and Windows (PC) i get only 900-1000 MBytes/sec - Result that i expected to be in ESXi
Tested speeds with iperf
What can be the problem? drivers?
By the way when i passthrough the card to FreeNAS, the FreeNAS doesn't recognize it.
Thank you for help
##Problem: Segmentation fault: invalid permissions for mapped object running mpi with CUDA
##Configurations
OS:
******************************
Centos 7.5 (3.10.0-862.el7.x86_64)
Connetivity:
******************************
Back to Back
Softwares:
******************************
cuda-repo-rhel7-9-2-local-9.2.88-1.x86_64
nccl_2.2.13-1+cuda9.2_x86_64.tar
MLNX_OFED_LINUX-4.3-3.0.2.1-rhel7.5-x86_64.tgz
nvidia-peer-memory_1.0-7.tar.gz
openmpi-3.1.1.tar.bz2
osu-micro-benchmarks-5.4.2.tar.gz
[root@LOCALNODE ~]# lsmod | grep nv_peer_mem
nv_peer_mem 13163 0
ib_core 283851 11 rdma_cm,ib_cm,iw_cm,nv_peer_mem,mlx4_ib,mlx5_ib,ib_ucm,ib_umad,ib_uverbs,rdma_ucm,ib_ipoib
nvidia 14019833 9 nv_peer_mem,nvidia_modeset,nvidia_uvm
[root@LOCALNODE ~]#
## Steps Followed
Followed document : http://www.mellanox.com/related-docs/prod_software/Mellanox_GPUDirect_User_Manual_v1.5.pdf
Openmpi command: mpirun --allow-run-as-root -host LOCALNODE,REMOTENODE -mca btl_openib_want_cuda_gdr 1 -np 2 -mca btl_openib_if_include mlx5_0:1 -mca -bind-to core -cpu-set 23 -x CUDA_VISIBLE_DEVICES=0 /usr/local/libexec/osu-micro-benchmarks/mpi/pt2pt/osu_latency -d cuda D D
## Two issues/problem seen where we need help from MNLX
1. While running osu micro benchmarks between Device to Device (i.e D D ) getting segmentation fault.
2. Though normal RDMA traffic (ib_send_*) is running fine between both the Nodes and on Both the Ports, But while running osu micro benchmarks, traffic is only going through Port 1 (MLX5_1)
Note: NVidia GPU and Mellanox Adapter are in different NUMA Nodes.
[root@LOCALNODE ~]# cat /sys/module/mlx5_core/drivers/pci\:mlx5_core/0000\:*/numa_node
1
1
[root@LOCALNODE ~]# cat /sys/module/nvidia/drivers/pci\:nvidia/0000\:*/numa_node
0
[root@LOCALNODE ~]# lspci -tv | grep -i nvidia
| +-02.0-[19]----00.0 NVIDIA Corporation GP100GL [Tesla P100 PCIe 16GB]
[root@LOCALNODE ~]# lspci -tv | grep -i mellanox
-+-[0000:d7]-+-02.0-[d8]--+-00.0 Mellanox Technologies MT27800 Family [ConnectX-5]
| | \-00.1 Mellanox Technologies MT27800 Family [ConnectX-5]
## Issue Details:
******************************
Issue 1:
[root@LOCALNODE nccl-tests]# mpirun --allow-run-as-root -host LOCALNODE,REMOTENODE -mca btl_openib_want_cuda_gdr 1 -np 2 -mca btl_openib_if_include mlx5_0 -mca -bind-to core -cpu-set 23 -x CUDA_VISIBLE_DEVICES=0 /usr/local/libexec/osu-micro-benchmarks/mpi/pt2pt/osu_latency -d cuda D D
--------------------------------------------------------------------------
No OpenFabrics connection schemes reported that they were able to be
used on a specific port. As such, the openib BTL (OpenFabrics
support) will be disabled for this port.
Local host: LOCALNODE
Local device: mlx5_0
Local port: 1
CPCs attempted: rdmacm, udcm
--------------------------------------------------------------------------
# OSU MPI-CUDA Latency Test v5.4.1
# Send Buffer on DEVICE (D) and Receive Buffer on DEVICE (D)
# Size Latency (us)
0 1.20
[LOCALNODE:5297 :0:5297] Caught signal 11 (Segmentation fault: invalid permissions for mapped object at address 0x7fd69ea00000)
==== backtrace ====
0 0x0000000000045e92 ucs_debug_cleanup() ???:0
1 0x000000000000f6d0 _L_unlock_13() funlockfile.c:0
2 0x0000000000156e50 __memcpy_ssse3_back() :0
3 0x00000000000318e1 uct_rc_mlx5_ep_am_short() ???:0
4 0x0000000000027a5a ucp_tag_send_nbr() ???:0
5 0x0000000000004c71 mca_pml_ucx_send() ???:0
6 0x0000000000080202 MPI_Send() ???:0
7 0x0000000000401d42 main() /home/NVIDIA/osu-micro-benchmarks-5.4.2/mpi/pt2pt/osu_latency.c:116
8 0x0000000000022445 __libc_start_main() ???:0
9 0x000000000040205b _start() ???:0
===================
-------------------------------------------------------
Primary job terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
-------------------------------------------------------
--------------------------------------------------------------------------
mpirun noticed that process rank 0 with PID 0 on node LOCALNODE exited on signal 11 (Segmentation fault).
--------------------------------------------------------------------------
[LOCALNODE:05291] 1 more process has sent help message help-mpi-btl-openib-cpc-base.txt / no cpcs for port
[LOCALNODE:05291] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages
[root@LOCALNODE nccl-tests]#
Issue 2:
[root@LOCALNODE ~]# cat /sys/class/infiniband/mlx5_0/ports/1/counters/port_*
0
0
0
0
0
0
0
0
0
0
0
[root@LOCALNODE ~]# cat /sys/class/infiniband/mlx5_1/ports/1/counters/port_*
0
18919889
0
1011812
0
0
0
9549739941
0
35318041
0
[root@LOCALNODE ~]#
Thanks & Regards
Ratan B
Hello, I would like to know which are the compatible InfiniBand cards with the Supermicro 1028R-TDW serverboard.
More specifically, I would also like to know if anyone knows about compatibility of this serverboard with the MT25408 InfiniBand cards.
Best Regards!
Andrés.
As I wrote above, I have manually set up irq in the virtual machine. You can see that there is only one queue in the hardware queue. I guess the driver does not support SR-IOV multi-queue function in the virtual machine.
VM：
[root@host-01 ~]# ls -la /sys/devices/pci0000\:00/0000\:00\:04.0/net/ib0/queues/
total 0
drwxr-xr-x 4 root root 0 Jun 29 10:11 .
drwxr-xr-x 5 root root 0 Jun 29 10:11 ..
drwxr-xr-x 2 root root 0 Jun 29 10:11 rx-0
drwxr-xr-x 3 root root 0 Jun 29 10:11 tx-0
Guest Host：
[root@testserver-1 ~]# ls -la /sys/devices/pci0000\:80/0000\:80\:01.0/0000\:81\:00.0/net/ib0/queues/
total 0
drwxr-xr-x 35 root root 0 Jun 28 19:59 .
drwxr-xr-x 5 root root 0 Jul 10 10:51 ..
drwxr-xr-x 2 root root 0 Jun 28 19:59 rx-0
drwxr-xr-x 2 root root 0 Jun 28 19:59 rx-1
drwxr-xr-x 2 root root 0 Jun 28 19:59 rx-10
drwxr-xr-x 2 root root 0 Jun 28 19:59 rx-11
drwxr-xr-x 2 root root 0 Jun 28 19:59 rx-12
drwxr-xr-x 2 root root 0 Jun 28 19:59 rx-13
drwxr-xr-x 2 root root 0 Jun 28 19:59 rx-14
drwxr-xr-x 2 root root 0 Jun 28 19:59 rx-15
drwxr-xr-x 2 root root 0 Jun 28 19:59 rx-2
drwxr-xr-x 2 root root 0 Jun 28 19:59 rx-3
drwxr-xr-x 2 root root 0 Jun 28 19:59 rx-4
drwxr-xr-x 2 root root 0 Jun 28 19:59 rx-5
drwxr-xr-x 2 root root 0 Jun 28 19:59 rx-6
drwxr-xr-x 2 root root 0 Jun 28 19:59 rx-7
drwxr-xr-x 2 root root 0 Jun 28 19:59 rx-8
drwxr-xr-x 2 root root 0 Jun 28 19:59 rx-9
drwxr-xr-x 3 root root 0 Jun 28 19:59 tx-0
drwxr-xr-x 3 root root 0 Jun 28 19:59 tx-1
drwxr-xr-x 3 root root 0 Jun 28 19:59 tx-10
drwxr-xr-x 3 root root 0 Jun 28 19:59 tx-11
drwxr-xr-x 3 root root 0 Jun 28 19:59 tx-12
drwxr-xr-x 3 root root 0 Jun 28 19:59 tx-13
drwxr-xr-x 3 root root 0 Jun 28 19:59 tx-14
drwxr-xr-x 3 root root 0 Jun 28 19:59 tx-15
drwxr-xr-x 3 root root 0 Jun 28 19:59 tx-16
drwxr-xr-x 3 root root 0 Jun 28 19:59 tx-2
drwxr-xr-x 3 root root 0 Jun 28 19:59 tx-3
drwxr-xr-x 3 root root 0 Jun 28 19:59 tx-4
drwxr-xr-x 3 root root 0 Jun 28 19:59 tx-5
drwxr-xr-x 3 root root 0 Jun 28 19:59 tx-6
drwxr-xr-x 3 root root 0 Jun 28 19:59 tx-7
drwxr-xr-x 3 root root 0 Jun 28 19:59 tx-8
drwxr-xr-x 3 root root 0 Jun 28 19:59 tx-9