Hi,
i want to build a Mellanox IP Conenction between my Freenas and Proxmox Server.
Both Servers have dual Port MHQH29-XTC Cards inside.
I try:
modprobe mlx4_en
echo eth > /sys/bus/pci/devices/0000\:0d\:00.0/mlx4_port1
echo eth > /sys/bus/pci/devices/0000\:0d\:00.0/mlx4_port2
set a ip Adress
ifup the interface
On Freeenas i see the interface is active, but i can not ping the other server.
Hello everyone,
I am planning to install two ConnectX-4 Lx EN network interface card, 25GbE dual-port SFP28, PCIe3.0 x8, tall bracket, ROHS R6 (MCX4121A-ACAT) in two Dell PowerEdge R540 servers. I want to the servers to work as a cluster with Hyper-V virtual machines and to replicate data from servers storage between them. Can anyone confirm whether the direct back-to-back connection between both servers without any switch inbetween using these Mellanox cards and two SFP28 cables will be possible and supported?
Thanks in advance!
I have a FPGA board from Xilinx. It has 3 SFP28 ports. When I use a DAC cable (MCP2M00-A002) to connect 2 of the ports, the ports come up. But when I use your AOC (MFA2P10-A003), the ports won't be up. Why?
Hi,
I'm having speed issues that I've isolated to my IS5030 switch.
System: I have two Infiniband switches: 1. A Mellanox IS50XX with 36 ports enabled and the FrabricIT internal subnet manager running, making it an IS5030, with latest firmware (IBM P/N: 98Y3756) 2. A Sun 36 port QDR Infiniband Switch, internal subnet manager. I also have 3 types of HCAs: Sun 375-3696 X4242A (rebranded Mellanox ConnectX-2 card), HP 544FLR-QDR (based on Mellanox ConnectX-3), and Intel/QLogic QLE7340. I haven't figured out how to update the firmware for the Sun products. The HP HCAs have the latest firmware. The QLE7340's don't use firmware. Running CentOS 7 on all nodes.
I've tried the following tests:
I had a long discussion with an Intel rep, and it turns out that these two Infiniband switches are not compatible with the QLE7340's (some of the later Mellanox X series switches are, though), which is probably why they aren't reaching QDR speeds. So let's ignore those for now.
The really weird thing is that the Sun and HP HCAs should really be negotiating to 40 Gb/s with the IS5030, and I have no idea why they are not. Clearly the HCAs, software, and cables (I tested all of these) are fine since they work at 40 Gb/s back-to-back.
Any ideas?
Ok, last reply on this topic. It's not 100% solved, but it's better.
See above for how to upgrade the firmware for the IS5031/IS5030. That's easy and solved.
The speed issues are a separate problem: I will create a second post about them, so ignore all of that here. Basically, it turns out that some of these IS50XX are actually DDR switches...be careful when buying.
The license problems are trickier. If you have the license(s) written on the bottom of the pull out tab, you're golden. If you don't, technical support can't help you, though. I tried contacting sales, but Mellanox's official policy (despite my begging) is to not support EoL hardware. Apparently generating the licenses for both the 36 port enable and the FabricIT (which allows you to use the subnet manager) from the serial number is a pain. They won't sell them to you, and have no plans for making them available. If you do manage to get one somehow, the FabricIT user guide shows how to install them, and that's very easy to do. Thus, I STRONGLY SUGGEST YOU ONLY BUY THE SWITCHES THAT HAVE THE LICENSES INSTALLED ALREADY. Ask the seller, make very sure that they come with both the 36 port enable license and the FabricIT (internal subnet manger) licenses. Offer to send them the console cable and instructions on how to check, whatever it takes. And when you get your switch, do NOT perform a hardware reset. First, go into console-> config mode, and type "show licenses" and write down the license keys (I think they're displayed there in full...you should be able to find them somewhere in there). They are long strings of letters/numbers and dashes. Then you can perform a hardware reset if you need to.
Hope all of this helps someone in the future...
I posted on servethehome, too, and learned some stuff from those guys.
I've done a lot of port configuration attempts with the webui and CLI. It says that the ports are capable of "2.5, 5 or 10Gbps", and 10Gbps 4x is QDR, but then none of the ports negotiate to more than 5 4x = 20 Gbps. I attempted to force the port speed (both the switch and the hca port) using ibportstate on the node, but trying to force the switch port resulted in an error and trying to force the hca then forcing a renegotiate (pull cable, reinsert cable) caused the link to never come up again until after a reboot.
It turns out that there are DDR only versions of the IS50XX switches. https://cw.infinibandta.org/files/showcase_product/101213.173410.167.IS5030_35.pdf I didn't think there were because ALL of the IS50XX switches are advertised as QDR, including the one I bought. Mine is an IBM rebrand, so unfortunately doesn't have the complete mellanox part number anywhere, but if yours has a "D" in the complete part number, then it's a DDR switch. "Q" means QDR. I'm not sure if this applies to me or not yet.
Hi All,
I have a remote system in which ib0,ib1,ib2,ib3,ib4 and ib5 are the infiniband network interfaces.
ib0 and ib1 are externally connected back to back with a cable[loopback connection].
ib2 and ib3 are not connected.
ib4 and ib5 are externally connected back to back with a cable[loopback connection].
Is there any Linux command or mellanox utility through which one can detect that [ ib0 - ib1 ] and [ ib4 - ib5 ] are the pair of network devices connected back to back given all the network devices? Please provide some pointers on how to detect the connected pair.
Thanks in Advance!
Hello, we reading this manual for vma:
http://www.mellanox.com/related-docs/prod_acceleration_software/VMA_8_4_10_User_Manual.pdf
For example there is "“Dummy Send” feature is supported in hardware starting from ConnectX-4 NIC"
and "Multi Packet Receive Queue: Mellanox ConnectX-4 LX / ConnectX-5 or above."
Will it work on connectx-4 lx? or only starts from connectx-4?
Which card "better"?
Thx for your cooperation, Alex.
ConnectX-5 provides roughly ~25% more performances than ConnectX-4 if using more than 1 queue with DPDK.
Also ConnectX-5 capable of packet header rewrite for open flow steering rules.
Thanks,
Amir
Hi,
so I'm setting up Scale-Out file server on Windows Server 2016 (fully patched for this date). When I use WinOF-2 newer then 1.70 (tried 1.80.50 and 1.80.51) The QoS is not working properly.
I'm using MXC416A-BCAT with firmware 12.21.2010.
I get RDMA to work but "WinOF-2 Port QoS" indicate that incorrect priorities are used and traffic is marked with wrong DSCP values. Also Get-NetAdapterQos indicates some kind of a problem.
My configuration:
Set-NetAdapterAdvancedProperty -Name SMB1 -RegistryKeyword "*JumboPacket" -RegistryValue 4200
Set-NetAdapterAdvancedProperty -Name SMB1 -RegistryKeyword VlanID -RegistryValue "4029"
Set-NetQosDcbxSetting -Willing 0 -Confirm:$false
Enable-NetQosFlowControl -Priority 4,5
Disable-NetQosFlowControl -Priority 0,1,2,3,6,7
New-NetQosTrafficClass "SMB" -Priority 5 -BandwidthPercentage 95 -Algorithm ETS
New-NetQosPolicy "SMBD" -NetDirectPortMatchCondition 445 -PriorityValue8021Action 5 -Confirm:$false -Precedence 255 -NetworkProfile All
New-NetQosPolicy "SMB" -SMB -PriorityValue8021Action 5 -DSCPAction 42 -Confirm:$false -Precedence 255 -NetworkProfile All
New-NetQosPolicy "DEFAULT" -Default -PriorityValue8021Action 5 -DSCPAction 42 -Confirm:$false -Precedence 240 -NetworkProfile All
New-NetQosPolicy "CLUSTER" -Cluster -PriorityValue8021Action 7 -DSCPAction 56 -Confirm:$false -Precedence 240 -NetworkProfile All
New-NetQosPolicy "TCP" -IPProtocolMatchCondition TCP -PriorityValue8021Action 0 -DSCPAction 0 -Confirm:$false -Precedence 10 -NetworkProfile All
New-NetQosPolicy "UDP" -IPProtocolMatchCondition UDP -PriorityValue8021Action 0 -DSCPAction 0 -Confirm:$false -Precedence 10 -NetworkProfile All
Enable-NetAdapterQos -Name SMB1
Set-NetAdapterQos -Name SMB1 -Enabled $true
$m=HKLM:\SYSTEM\CurrentControlSet\Control\Class\{4d36e972-e325-11ce-bfc1-08002be10318}\0001
new-itemProperty -Path $m -Name "RxTrustedState" -PropertyType "String" -Value "2" -Force
new-itemProperty -Path $m -Name "TxUntagPriorityTag" -PropertyType "String" -Value "1" -Force
new-itemProperty -Path $m -Name "PriorityToDscpMappingTable_7" -PropertyType "String" -Value "56" -Force
new-itemProperty -Path $m -Name "PriorityToDscpMappingTable_5" -PropertyType "String" -Value "42" -Force
new-itemProperty -Path $m -Name "PriorityToDscpMappingTable_4" -PropertyType "String" -Value "34" -Force
Mlx5Cmd.exe -Qosconfig -Dcqcn -Name SMB1 -Set -DcqcnCnpDscp 58 -DcqcnCnp802pPrio 7
Mlx5Cmd.exe -Qosconfig -Dcqcn -Name SMB1 -Disable
Mlx5Cmd.exe -Qosconfig -Dcqcn -Name SMB1 -Enable 4
Mlx5Cmd.exe -Qosconfig -Dcqcn -Name SMB1 -Enable 5
#Mlx5Cmd.exe -Qosconfig -Dcqcn -Name "SMB1" -Set –DcqcnCnpPrioMode 0 #btw. this is not working - Invalid command or flag,
Restart-NetAdapter SMB
Enable-NetAdapterRDMA SMB1
So I want to see SMB Direct on priority 5 with DSCP 42. However counters indicate priority 3 is used instead of 5.
So I'm checking:
PS C:\Windows\system32> Get-NetAdapterqos SMB1 Name : SMB1 Enabled : True Capabilities : Hardware Current -------- ------- MacSecBypass : NotSupported NotSupported DcbxSupport : IEEE IEEE NumTCs(Max/ETS/PFC) : 1/1/1 1/1/1
This is clearly wrong.
Then I reboot the server. And get the same result.
Then I check QoSPolicy:
PS C:\Windows\system32> Get-NetQosPolicy -PolicyStore "ActiveStore" | select elementname,name,owner,NetworkProfile,Precedence,IPProtocol,Template,PriorityValue,DSCPValue,DSCPAction,NetDirectPort | sort -Descending Precedence,elelementname | ft elementname name owner NetworkProfile Precedence IPProtocol Template PriorityValue DSCPValue DSCPAction ----------- ---- ----- -------------- ---------- ---------- -------- ------------- --------- ---------- livemigration livemigration Group Policy (Machine) All 255 None LiveMigration 4 34 34 smb smb Group Policy (Machine) All 255 None SMB 5 42 42 smbd smbd Group Policy (Machine) All 255 None None 5 -1 -1 cluster cluster Group Policy (Machine) All 240 None Cluster 7 56 56 default default Group Policy (Machine) All 240 None Default 5 42 42 tcp tcp Group Policy (Machine) All 10 TCP None 0 0 0 udp udp Group Policy (Machine) All 10 UDP None 0 0 0
So policy is ok. But traffic is using priority 3 and DSCP 26 - and these values are not configured anywhere!
When I use driver 1.70 priorites are used correctly.
Any one has any ideas what can I do to get priorities working correctly with current driver version ?
Thanks for your answer. The documentation doesn't say much about the differences and why X-5 is better for NFV use cases.
Hi Alexander,
you are referring to two different features:
These two features are used for different use cases.
The Dummy send is been used for low rate traffic - usually in the financial market.
The Multi Packet Receive Queue is been used for Media use cases.
I hope this answers your question.
Best Regards,
Nir Nitzani
Hello, Nir.
Thanks for your answer.
So does it mean that Explicit Ring Polling doesnt supported by ConnectX-4LX ?
Hi Alex,
If you are trying to use Multi Packet Receive Queue with VMA API it's focused on Media use cases.
I think it will be best if you be more specific with your use case and needs and we can direct you how it can be achieved.
Thanks,
Nir
Hi Nir,
It's for market, but we want try Multi Packet Receive Queue too.
Can you answer about Explicit Ring Polling?
We interested in it much more than i MPRQ
Hi Alex,
In order to get data directly from the Ring you can try our SocketXtreme feature.
Hi Nir,
I cant find info about working it on X-4 or X-4LX, there is only vma manual for 8.4.* version, not higher.
Alex,
please refer the VMA user manual: http://www.mellanox.com/related-docs/prod_acceleration_software/VMA_8_4_10_User_Manual.pdf
Okey Nir,
Thank a lot for your cooperation
Hi Justin,
Please note that the connectX-2 adapter is reached its end of life and end of support period following that it was not certified against the latest kernel versions.
Best Regards,
Viki