Basically by validating (qp_access_flags. buffer sizes etc.) I mean, I made sure they have the right values. For example qp_access flags was enabled for RDMA read and write,
buffer size in the work request matched with that used for memory registration and so on
I was able to get this issue resolved. The problem was with the "max_dest_rd_atomic" QP attribute. Per documentation, "max_dest_rd_atomic" is "number of RDMA Reads outstanding at any time for this QP as a destination". Our code was using RDMACM for connection management. The way "max_dest_rd_atomic" is set by RDMACM is via attribute called "responder_resources" sent as an argument "rdma_conn_param" to "rdma_connect". The argument did not look obvious and hence was not set causing RDMACM to set "max_dest_rd_atomic" to zero. causing RDMA reads initiated to this node to fail.
Basically the syndrome "Remote_Invalid_Request_Error" means lot of issues that are not clearly defined, hence it took us time to figure out the exact issue. This is where I was hoping that "vendor syndrome" might come in handy to figure out root cause for "Remote_Invalid_Request_Error" or similar errors that have multiple failure reasons. Unfortunately "vendor syndrome" doesn't seem to be exported by Mellanox. It will help if Mellanox could export this error with its corresponding description such that it will help Mellanox RDMA users to debug similar issues.
Thanks !
Description of problem:
When a node reboots, pacemaker starts before IB SRP LUNs are accessible which causes clustered VG and NFS resource residing on these LUNs fail, and we have to stop and start pacemaker service manually to bring those resources back online.
Version-Release number of selected component (if applicable):
OS:RHEL7.2(default kernel 3.10.0-327.el7.x86_64) with high-availability and resilient storage add-on
IB HBA model:Mellanox ConnectX-3 VPI
IB HBA driver:MLNX_OFED_LINUX-3.4-2.0.0.0
IB HBA firmware:v2.40.7000
Storage array: IBM FlashSystem-900
How reproducible:
always
Steps to Reproduce:
1.create two-node RHCS7.2 cluster framework
2.create clustered volume group and logical volume/ Filesystem/NFS/VIP resource
3.reboot one node
Actual results:
From system log we can see clearly that Infiniband SRP LUNs are still inaccessible when pacemaker service starts, as such Filesystem resource residing on clustered VG/LV and NFS resource all failed.
Expected results:
IB SRP LUNs should be accessible first before pacemaker starts, then Filesystem resource residing on clustered VG/LV and NFS resource can start successfully.
Additional info:
Host is accessing storage through Infiniband SRP(SCSI RDMA Protocol)
Good morning,
Could you kindly help me find out how old this card in our Linux machine?
Our server is very slow in running MPP jobs.
lshal | grep system.hardware
system.hardware.product = 'C2112-4GP3' (string)
system.hardware.vendor = 'SGI.COM' (string)
system.hardware.version = '0123456789' (string)
Is this system really considered old already?
Command; lspci | grep Mellanox
Return; 02:00.0 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3]
lspci -vv -s 02:00.0
02:00.0 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3]
Subsystem: Mellanox Technologies Device 0149
Physical Slot: 5
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisIN Tx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INT x-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 32
Region 0: Memory at c7200000 (64-bit, non-prefetchable) [size=1M]
Region 2: Memory at c5800000 (64-bit, prefetchable) [size=8M]
Expansion ROM at c7100000 [disabled] [size=1M]
Capabilities: <access denied>
Kernel driver in use: mlx4_core
Regards, Harry
Hello,
I'm using one ConnectX2 10G Card on my Server (FreeNAS) and one on my Windows 10 Desktop System.
But on my Windows 10 System, the ConnectX2 is shown in the "Safely Remove Hardware" tray, like a external Harddrive. Installed the newest Win 10 Driver (5.35) and even reinstalling Windows didn't help.
I'm using a lot external Drives, so it's really annoying to always restart the System after accidentally removing the Card. Tried it with Firmware 2.9.1000, 2.9.1200 and 2.10.720, but didn't helped.
Would be really great if someone could help me with this.
(Sorry for my bad english!)
logs:
send arp command: arping2 -D -q -c 1 -C 0 -w 1000 -I swid0_eth.11 -t 7C:FE:90:BE:A2:50 -s 24:8A:07:CD:04:08
kernel: [15706142.471406] sx_core 0000:03:00.0: triggering backup poll for CQ:26 RDQ:2 cons_index:172196233
kernel: [15706142.471425] sx_core: sx_eq_int: doing backup poll for QN:26
Please open a support case support@mellanox.com
Each ULP module registers an ib_client，When the CX5-ROCE card is loaded, you do not want the add () function to be called, or do nothing in the add () function, and when you load a CX5-IB card, you want to load it as expected。
Originally hoped that VERBS rdma_port_get_link_layer can identify the link layer for IB or ETH，however, the implementation of this interface to determine the type of link layer based on the type of transport layer，both RoCE(v1 and v2) and Infiniband use IBTA transport, anything was wrong about the VERBS?
Hello,
I have two servers each of which installed the latest Mellanox firmware and the Mellanox OFED 4.1 software stack. I tried the ib_send_bw on two machines, but get the error as it shows in following pictures. I have reinstalled the whole software stack for few times, but the problem still exist. Can someone please help me with this problem? Many thanks.
We have setup here, with x86 host has connectX3 Pro 314A card, and communicating with our target board that has 10G ethernet interface.
On the target side, we have RoCEv2 protocol stack over which NVMOF application is running.
On host side we are executing FIO read, for this host generates NVMe SSD read commands to the target. In turn, target generates RDMA Write requests and following that sends completions.
We have IBdump that is attached. The issue we are encountering is for RDMA Write request, host is sending Remote Access Error as ACKNOWLEDGEMENT. As per Infiniband Specification (section 9.7.5.2.4 REMOTE ACCESS ERROR), reasons for sending this response (with Remote Access Error) are following,
1) The R_Key field of the RETH is invalid.
2) The virtual address and length or type of access specified is outside the locally defined limits associated with the R_Key.
3) For an HCA, a protection domain violation is detected.
Analyzing the IB dump, we did not find that any of above violations. From our analysis,
QP#243 on host paired with QP#3 on target. For RDMA write request with PSN 9529773 from target, received response with Remote Access Error. The R_KEY is 75014, Virtual Address is 8150351872, and
DMA Length is 4096 bytes. The corresponding received NVME command capsule from host is with PSN 9529418. The RETH parameters (R_KEY, Virtual Address and DMA Length) are exactly matching as
specified in NVMe command.
We have also verified that there are no repeated R_KEY and Virtual Address pairs. We wanted to understand the reasons why host is issuing Remote Access Error. Please help us.
In order to update the firmware of OEM adapters from Dell (or any others for that matter), you must download their firmware release that matches the PSID of the adapter. In your case, from what you shared, the PSID of the adapter in question is DEL2180110032.
The best way to find the driver is to go to the Dell support page for your server using the service tag and locate the drivers and firmware packages there. None of the OEM adapters that are not Mellanox-branded may be updated with the OFED packages from mellanox.com.
This was the latest release I could find on the Dell site: Mellanox Linux Drivers and Install Script for Mellanox ConnectX-3, ConnectX-4 Lx and ConnectX-4 Ethernet adapters, Mella…
Hope this helps!
Hello,
i got some useful information from the older posters "https://community.mellanox.com/docs/DOC-2086#comment-2056 "
The problem was because of multiple network cards installed on one server, when i manually specified the network card, the problem is gone.
Hello,
I am very interested in Innova Flex 4 Lx EN Adapter Cards - http://www.mellanox.com/related-docs/prod_adapter_cards/PB_Innova_Flex4_Lx_EN.PDF
I need to implement a simple pattern matching solution for x86 platform (REGEX/DPI), actually in the starting phase of the project to examine SNI fields present in the packet payload during TLS handshake.
So I am looking for some Smart NIC capable of doing regex operations (DPI) on the packets that are passing through it. I hope Innova Flex (or others?) should be capable of doing it (according to a datasheet).
So, the question is presence of the NIC drivers for Linux and support for the pattern matching. Ideally I need a proprietary iptables module that would handle the traffic matching with your NIC.
Does it already exist or is there a SDK for Linux iptables with some example code (i.e. how to implement a iptables module) how to consume hardware regex offload capabilities of Smart NICs ?
I need to have something like iptables modules called xt_layer7 or xt_string, but L7/REGEX matching to done by hardware offload...
Thank you for any answer.
Jan
Thank you, Joseph!
On a CentOS bases system i want to replace/update the Inbox driver version 2.2-1 with latest mlnx drivers.
System is equippped with ConnectX3 / MT27500.
downloaded latest package "MLNX_OFED_LINUX-4.2-1.0.0.0-rhel7.4-x86_64.tgz" but get errors when trying to install.
my system is using Kernel 4.12.4-1.el7.elrepo.x86_64
# ./mlnxofedinstall
Current operation system is not supported!
# modinfo mlx4_core | head
filename: /lib/modules/4.12.4-1.el7.elrepo.x86_64/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko
version: 2.2-1
license: Dual BSD/GPL
description: Mellanox ConnectX HCA low-level driver
author: Roland Dreier
What's the recommended way to update/replace/ the RHEL/CentOS Inbox driver (mlx4_core, mlx4_en) ?
Both ports run in ETH-mode, i do not use IB.
This has been on my mind for a while. I am sure this have been covered somewhere, but I have been reading for the past hour and have not come up with an answer.
Is there a difference between QDR 40Gb and FDR 56Gb cables?
I have ConnectX-3 Cards and SMX6036 Switch, but they are syncing up at 40Gbs. I am using HP Fiber optic cables HP 498386-B24. The cables are marked as QDR.
Going forward I wanted to know if I needed to purchase cables specifically marked for FDR, and if so how do I distinguish. It seems cables are usually marked as QSFP+ or DDR or QDR. Not many mention FDR.
Thanks,
Todd
FDR cable also tagged 56Gb in blue color.
If you have a SX6036F FDR14 Infiniband switch, you also access User Guide and Release Notes on web management UI.
These guides also have several informations supported cables that distinguished with part number.
BR,
Jae-Hoon Choi
When I attempt to install the driver for this card, I get the following error:
Uninstalling the previous version of mlnx-en
Failed to uninstall the previous installation
Collecting debug info...
See /tmp/mlnx-en-4.1-1.0.2.0.28167.logs/ofed_uninstall.log
----------------------------------------------------------------------------------------------------------
lspci output shows the card:
a0:00.0 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3] (rev ff)
----------------------------------------------------------------------------------------------------------
Also if I try to update the firmware, I get the following error:
Infiniband : card detection : DETECTED FDR Mellanox
Infiniband : card position : DETECTED @ a0:00.0
Infiniband : mlx FDR detection : PASSED
Infiniband : mlx FDR revision : ABORTING! (/usr/bin/mlxburn is missing)
FDR Infiniband : Firmware expected 2.32.5100 : Can't Detect current firmware
Infiniband : mlx FDR FW flash file : Can't find:
Firmware update not possible.
/usr/bin/mlxburn does exist though
I use cma_roce_mode to modify gid 0 roce v1 to roce v2, but I doesn't work. the gid's info are listed. Can anybody tell me how to do?
Hi，I have do as this guide said. But my ceph cluster is health_err. The error information is "HEALTH_ERR 320 pgs are stuck inactive for more than 300 seconds; 320 pgs stuck inactive; 320 pgs stuck unclean".
In ceph ms log, the rdma information(gid\lid\qpn) that are send by the osd task are right. But the rdma information(gid\lid\qpn) that are received by the osd task are 0.
Ceph log:
Infiniband send_msg sending: 3, 1321022, 0, 0, fe80000000000000248a070300f8cd01
Infiniband recv_msg recevd: 0, 0, 0, 0, ▒̽V▒▒
sudo ceph daemon osd.0 perf dump AsyncMessenger::RDMADispatcher
{
"AsyncMessenger::RDMADispatcher": {
"polling": 0,
"rx_bufs_in_use": 0,
"rx_bufs_total": 8192,
"tx_total_wc": 9514,
"tx_total_wc_errors": 9514,
"tx_retry_errors": 4759,
"tx_wr_flush_errors": 4755,
"rx_total_wc": 0,
"rx_total_wc_errors": 0,
"rx_fin": 0,
"handshake_errors": 0,
"total_async_events": 0,
"async_last_wqe_events": 0,
"created_queue_pair": 5040,
"active_queue_pair": 4
}
}