Hi Francois,
Please check https://community.mellanox.com/docs/DOC-2964
and unmerged yet patchset https://review.openstack.org/#/c/504911/
↧
Re: Running ASAP2
↧
Re: Melanox grid director 4036e won't boot.
Hello Rav,
Mellanox do not seel servicecontracts on 4036E switches anymore. The product is at EOL stage.
for more information, please refer to our EOL info page at: http://www.mellanox.com/page/eol
Sorry we couldn't assist you.
Thanks
↧
↧
Re: Running ASAP2
Hi Lenny
I can’t access this document.
Can you please send me the PDF version.
There seems to be something wrong with my Mellanox account.
Regards
Francois Kleynhans
↧
Re: Running ASAP2
I also cannot access the documents listed...
↧
Re: How to configure MCX354A-FCBT Mellanox InfiniBand speed at 56Gbps ?
From the output prints you have presented, it looks like your SX6036 switch should be good & supports 56Gb (FDR)
- cables & nics are also fine and capable of fdr
- cables present: Infiniband speeds : SDR , DDR , QDR , FDR
- switch presents:
Supported LLR speeds : FDR10, FDR - which indicates you can set it to 56Gb
Supported speeds : sdr, ddr, qdr, fdr10 - here you see that fdr is missing
so, in my view - all you have to do is to run the following command on the switch that will add FDR on "supported speeds" list
run on CLI commnd:
(config) # interface ib <interface#> speed sdr ddr qdr fdr10 fdr force
(config) # configure write (to save the changes)
you should now see: Supported speeds : sdr, ddr, qdr, fdr10, fdr
this should enable you to uses 56Gb on the switch and on the nics as well
↧
↧
Re: Running ASAP2
Please try now.
↧
Re: Running ASAP2
Thanks I can access it now
↧
Re: NVMeOF SLES 12 SP3 : Initiator with 36 cores unable to discover/connect to target
Hi,
Adding the parameter didn't help. It still gives the same error:
athena:~ # nvme discover -t rdma -a 192.168.0.1 -s 4420
Failed to write to /dev/nvme-fabrics: Invalid argument
athena:~ # dmesg |tail -1
[ 1408.720843] nvme_fabrics: unknown parameter or missing value 'hostid=a61ecf3f-2925-49a7-9304-cea147f61ae' in ctrl creation request
athena:~ # nvme connect -t rdma --nr-io-queues=32 -a 192.168.0.1 -s 4420 -n mcx
Failed to write to /dev/nvme-fabrics: Invalid argument
athena:~ # !dm
dmesg |tail -1
[ 1437.914081] nvme_fabrics: unknown parameter or missing value 'hostid=a61ecf3f-2925-49a7-9304-cea147f61ae' in ctrl creation request
↧
Re: How to configure MCX354A-FCBT Mellanox InfiniBand speed at 56Gbps ?
HPE Part Number 712498-B21 is SX6036T FDR10 switch.
Therefore in this case user can't use FDR14 56Gb speed in fabric.
HPE Support document - HPE Support Center
Why did you give a incorrect information to community users?
Best Regard,
Jae-Hoon Choi
↧
↧
Is VMA really use RDMA to implement offload?
Hello, I'm learning RDMA and testing VMA. In the VMA user manual, it says that VMA implements native RDMA verbs API. And in the source code of VMA, I really see the RDMA verbs. However, when I run VMA with LD_PRELOAD and VMA_SPEC=latency, it runs slower than TCP. I add a log before the ibv_post_send function invocation and recompile the code. The log does not show up! And in the source code, I see the enum type below
enum tcp_sock_offload_e {
TCP_SOCK_PASSTHROUGH = 1, // OS handling this socket connection
// TCP_SOCK_RDMA_CM, // Offloaded, uses RDMA CM - SDP like connection
TCP_SOCK_LWIP // Offloaded, uses LWIP for wire compatible TCP impl
};
The TCP_SOCK_RDMA_CM is annotated! But it is also true that VMA implements the RDMA verbs. I am confused that which VMA really uses to implement offload, RDMA or LWIP? Or do I miss some configuration parameters that are necessary when using RDMA in VMA?
I test the VMA on the machine with ConnectX®-5 and the OFED 4.1 is installed.
↧
Re: LAG problems
Hi Rick,
LAG should work fine b/w SX1036 and the SN2700 switch. Only for MLAG we have the limitations of the cpu which should match for both the switches.
Can you please verify your configs?
Is this a regular LACP port channel b/w both the switches.
What is the status of the second port which you are bundling in a LACP? Is it up/down/suspended?
Please share me the details.
Thanks
Khwaja
↧
Why are IB Interfaces are running at 20gbps with Unexpected actual link speed 5
TLDR: How do I resolve this error:
Unexpected actual link speed 5 (enable_speed1="2.5 or 5 or 10", enable_speed2="2.5 or 5 or 10" therefore final speed should be 10)
I am not sure if I am missing a step somewhere, but the fabric works just with very high ping times
C:\> ibdiagnet
...
Speed / Width checks
-I- Link Speed Check (Compare to supported link speed)
-E- Links Speed Check finished with errors
-E- Link: S24be05ffffcb19e3/N24be05ffffcb19e0(ECHO2)/P1<-->Sf4521403002220f0/Nf4521403002220f0(SwitchX - Mellanox Technologies)/P3 - Unexpected actual link speed 5 (enable_speed1="2.5 or 5 or 10", enable_speed2="2.5 or 5 or 10" therefore final speed should be 10)
-E- Link: Sf4521403002220f0/Nf4521403002220f0(SwitchX - Mellanox Technologies)/P1<-->S24be05ffffcbb733/N24be05ffffcbb730(ECHOHCTL)/P1 - Unexpected actual link speed 5 (enable_speed1="2.5 or 5 or 10", enable_speed2="2.5 or 5 or 10" therefore final speed should be 10)
-E- Link: Sf4521403002220f0/Nf4521403002220f0(SwitchX - Mellanox Technologies)/P2<-->S24be05ffffaa8853/N24be05ffffaa8850(ECHO1)/P1 - Unexpected actual link speed 5 (enable_speed1="2.5 or 5 or 10", enable_speed2="2.5 or 5 or 10" therefore final speed should be 10)
-I- Link Width Check (Compare to supported link width)
-I- Links Width Check finished successfully
---------------------------------------------
Cable Diagnostic (Plugin)
-I- Cable Info retrieving finished successfully
---------------------------------------------
Summary
-I- Stage Warnings Errors Comment
-I- Discovery 0 0
-I- Lids Check 0 0
-I- Links Check 0 0
-I- Subnet Manager 0 0
-I- Port Counters 0 0
-I- Nodes Information 0 0
-I- Speed / Width checks 0 3
-I- Partition Keys 0 0
-I- Alias GUIDs 0 0
-I- Cable Diagnostic (Plugin) 0 0
***********************************************************************************
C:\> ibstat
CA 'ibv_device0'
CA type:
Number of ports: 2
Firmware version: 2.40.5030
Hardware version: 0x0
Node GUID: 0x24be05ffffcbb730
System image GUID: 0x24be05ffffcbb733
Port 1:
State: Active
Physical state: LinkUp
Rate: 20
Base lid: 6
LMC: 0
SM lid: 6
Capability mask: 0x90580000
Port GUID: 0x24be05ffffcbb731
Link layer: IB
Transport: IB
Port 2:
State: Down
Physical state: Polling
Rate: 40
Base lid: 0
LMC: 0
SM lid: 0
Capability mask: 0x90580000
Port GUID: 0x24be05ffffcbb732
Link layer: IB
Transport: IB
CA 'ibv_device0'
CA type:
Number of ports: 2
Firmware version: 2.40.5030
Hardware version: 0x0
Node GUID: 0x24be05ffffaa8850
System image GUID: 0x24be05ffffaa8853
Port 1:
State: Active
Physical state: LinkUp
Rate: 20
Base lid: 5
LMC: 0
SM lid: 6
Capability mask: 0x90580000
Port GUID: 0x24be05ffffaa8851
Link layer: IB
Transport: IB
Port 2:
State: Down
Physical state: Polling
Rate: 40
Base lid: 0
LMC: 0
SM lid: 0
Capability mask: 0x90580000
Port GUID: 0x24be05ffffaa8852
Link layer: IB
Transport: IB
CA 'ibv_device0'
CA type:
Number of ports: 2
Firmware version: 2.40.5030
Hardware version: 0x0
Node GUID: 0x24be05ffffcb19e0
System image GUID: 0x24be05ffffcb19e3
Port 1:
State: Active
Physical state: LinkUp
Rate: 20
Base lid: 7
LMC: 0
SM lid: 6
Capability mask: 0x90580000
Port GUID: 0x24be05ffffcb19e1
Link layer: IB
Transport: IB
Port 2:
State: Down
Physical state: Polling
Rate: 40
Base lid: 0
LMC: 0
SM lid: 0
Capability mask: 0x90580000
Port GUID: 0x24be05ffffcb19e2
Link layer: IB
Transport: IB
C:\>ibnetdiscover
#
# Topology file: generated on Fri Sep 29 11:29:00 2017
#
# Initiated from node 24be05ffffaa8850 port 24be05ffffaa8851
vendid=0x2c9
devid=0xc738
sysimgguid=0xf4521403002220f0
switchguid=0xf4521403002220f0(f4521403002220f0)
Switch 18 "S-f4521403002220f0" # "SwitchX - Mellanox Technologies" base port 0 lid 2 lmc 0
[1] "H-24be05ffffcbb730"[1](24be05ffffcbb731) # "ECHOHCTL" lid 6 4xDDR
[2] "H-24be05ffffaa8850"[1](24be05ffffaa8851) # "ECHO1" lid 5 4xDDR
[3] "H-24be05ffffcb19e0"[1](24be05ffffcb19e1) # "ECHO2" lid 7 4xDDR
vendid=0x2c9
devid=0x1003
sysimgguid=0x24be05ffffcb19e3
caguid=0x24be05ffffcb19e0
Ca 2 "H-24be05ffffcb19e0" # "ECHO2"
[1](24be05ffffcb19e1) "S-f4521403002220f0"[3] # lid 7 lmc 0 "SwitchX - Mellanox Technologies" lid 2 4xDDR
vendid=0x2c9
devid=0x1003
sysimgguid=0x24be05ffffcbb733
caguid=0x24be05ffffcbb730
Ca 2 "H-24be05ffffcbb730" # "ECHOHCTL"
[1](24be05ffffcbb731) "S-f4521403002220f0"[1] # lid 6 lmc 0 "SwitchX - Mellanox Technologies" lid 2 4xDDR
vendid=0x2c9
devid=0x1003
sysimgguid=0x24be05ffffaa8853
caguid=0x24be05ffffaa8850
Ca 2 "H-24be05ffffaa8850" # "ECHO1"
[1](24be05ffffaa8851) "S-f4521403002220f0"[2] # lid 5 lmc 0 "SwitchX - Mellanox Technologies" lid 2 4xDDR
C:\>ibping -L 6
Pong from ?hostname?.?domainname? (Lid 6): time 0.214 ms
Pong from ?hostname?.?domainname? (Lid 6): time 0.118 ms
Pong from ?hostname?.?domainname? (Lid 6): time 0.101 ms
Pong from ?hostname?.?domainname? (Lid 6): time 0.099 ms
Pong from ?hostname?.?domainname? (Lid 6): time 0.122 ms
Pong from ?hostname?.?domainname? (Lid 6): time 0.102 ms
Pong from ?hostname?.?domainname? (Lid 6): time 0.132 ms
Pong from ?hostname?.?domainname? (Lid 6): time 0.104 ms
Pong from ?hostname?.?domainname? (Lid 6): time 0.125 ms
Pong from ?hostname?.?domainname? (Lid 6): time 0.111 ms
Pong from ?hostname?.?domainname? (Lid 6): time 0.099 ms
Pong from ?hostname?.?domainname? (Lid 6): time 0.093 ms
Pong from ?hostname?.?domainname? (Lid 6): time 0.111 ms
Pong from ?hostname?.?domainname? (Lid 6): time 0.092 ms
Pong from ?hostname?.?domainname? (Lid 6): time 0.108 ms
Pong from ?hostname?.?domainname? (Lid 6): time 0.115 ms
Pong from ?hostname?.?domainname? (Lid 6): time 0.123 ms
--- ?hostname?.?domainname? (Lid 6) ibping statistics ---
17 packets transmitted, 17 received, 0% packet loss, time 18548 ms
rtt min/avg/max = 0.092/0.115/0.214 ms
↧
Re: INFINIBAND RDMA_CM_EVENT_ADDR_ERROR
Hi Sungho,
Thank you for posting your question on the Mellanox Community.
In your environment, when using multiple interfaces in the same address range, please bind the address on which you want to run udaddy/rping and / or ib_send_bw
For example:
rping - Server
# rping -d -s -a <ip-address-of-ib0>
rping - Client
# rping -d -c -a <ip-address-of-server>
udaddy - Server
# udaddy -b <ip-address-of-ib0>
udaddy - Client
# udaddy -b <ip-address-of-ib0> -s <ip-address-of-server>
ib_send_bw - Server
# ib_send_bw -d <ib-dev> -p <port> --report_gbits -R -a -F
Example: # ib_send_bw -d mlx5_0 -p 1 --report_gbits -R -a -F
ib_send_bw - Client
# ib_send_bw -d <ib-dev> -p <port> <IPoIB-of-server> --report_gbits -a -R -F
Example: # ib_send_bw -d mlx4_0 -p1 1.1.1.101 --report_gbits -a -R -F
In our lab, we have seen no issues running the above tests. All tests established and confirmed RDMA connectivity.
If you still experiencing issues, running the provided example, we recommend you to open a Support Case with Mellanox Technical Support.
Thanks.
Cheers,
~Martijn
↧
↧
Re: Trouble making Infiniband running udaddy
Hi Sungho,
Thank you for posting your question on the Mellanox Community.
In your environment, when using multiple interfaces in the same address range, please bind the address on which you want to run udaddy/rping and / or ib_send_bw
For example:
rping - Server
# rping -d -s -a <ip-address-of-ib0>
rping - Client
# rping -d -c -a <ip-address-of-server>
udaddy - Server
# udaddy -b <ip-address-of-ib0>
udaddy - Client
# udaddy -b <ip-address-of-ib0> -s <ip-address-of-server>
ib_send_bw - Server
# ib_send_bw -d <ib-dev> -p <port> --report_gbits -R -a -F
Example: # ib_send_bw -d mlx5_0 -p 1 --report_gbits -R -a -F
ib_send_bw - Client
# ib_send_bw -d <ib-dev> -p <port> <IPoIB-of-server> --report_gbits -a -R -F
Example: # ib_send_bw -d mlx4_0 -p1 1.1.1.101 --report_gbits -a -R -F
In our lab, we have seen no issues running the above tests. All tests established and confirmed RDMA connectivity.
If you still experiencing issues, running the provided example, we recommend you to open a Support Case with Mellanox Technical Support.
Thanks.
Cheers,
~Martijn
↧
problem about [ConnectX-4 Lx] card after upgrading
Hi,
I have a Dell R530 server, and I update the latest ConnectX-4 Lx firmware with OEM BIN file downloaed from Dell support web site.
http://www.dell.com/support/home/cn/zh/cndhs1/product-support/servicetag/4fp3xj2/drivers
I also installed the OFED downed from Mellanox, which is MLNX_OFED_LINUX-4.1-1.0.2.0-rhel7.4-x86_64
I reboot the system, and try the OFED stack with ib_send_bw which prints out some error message as follows:
the os I have is CentOS 7.4. I have spend some time on this problem, but get no hints. I may want to ask and get some help here.
thanks in advance.
↧
Re: NFS over RoCE Ubuntu 16.04 with latest OFED
I had the same in Centos 7 environment with OFED 3.3.x.x , I know that Mellanox is not supported NFS over RDMA with OFED 3.4 onwards but 3.3 in my situation is also don't working. Maybe Mellanox support will tell us more about it ??
BR
Adam
↧
ESXi 6.5 update 1 inbox driver - 56GbE speed support with CX-3 & SX6036G
Hi!
I tried ESXi 6.5 update 1 inbox driver 56GbE link between CX-3 (not Pro) and SX6036G Ethernet port.
But ESXi host shows me a link down, only link up with SX6036G 40GbE port mode.
Here is a link about it.
HowTo Setup 56GbE Back-to-Back on two servers
I can download latest CX-3 firmware bin format, but mlx format can't.
How can I resole this issue?
BR,
Jae-Hoon Choi
↧
↧
Hpe 5950 dac cable to mellanox MCX4121A-ACAT
I have an Hpe 5950 with sfp28 card. From server side i have melanox nic sfp28 dual port (MCX4121A-ACAT). What dac cable should i use and reference ? Anyone tested this config or any certification paper ?
↧
Re: NFS over RDMA on OEL 7.4
Hi Adam,
Indeed, NFSoRDMA (NFS over RDMA) is no longer supported by MLNX_OFED since driver version 3.4.
If you are using the Inbox Driver provided by the OS (RHEL 7.4), we Mellanox support and/or assistance is on a best effort basis due to the Inbox driver support being handle through the OS vendor.
Why? The driver which comes with the RHEL OS is derived from the upstream kernel. The OS Vendor gets the driver from kernel.org and modifies the code to their own needs.
We do not control that code nor do we know what modification the vendor has done to that code.
The versions are also not the same between Mellanox OFED driver and INBOX driver.
That being said, a few recommendations and suggestions below:
Make sure HCA's FW are aligned and to the latest revision (Mellanox.com)
Make sure Switche(s) MLNX_OS are aligned and t the latest revision (Mellanox.com)
You can consult:
A) Performance Tuning for Mellanox Adapters
Performance Tuning for Mellanox Adapters
B) Red Hat Enterprise Linux Network Performance Tuning Guide
Red Hat Enterprise Linux Network Performance Tuning Guide
Performance Tuning Guide (Link)
D) How To Configure LACP on Mellanox Switches
HowTo Configure LACP on Mellanox Switches
E) Troubleshoot LAG/MLAG LACP-PDU Rate Issues
Troubleshoot LAG/MLAG LACP-PDU Rate Issues
F) Perftest Package (RDMA)
Sophie.
↧
Re: problem about [ConnectX-4 Lx] card after upgrading
Hello, Hui,
did you try to contact Dell support?
Also, do note (based on the screen capture above) that the card is at Ethernet mode. If you need InfiniBand support, you need to change the port type to IB.
You will need to install MFT (or in case you have MOFED it is already there) and use: /sbin/connectx_port_config
Cheers; hope it helped..
↧