Quantcast
Channel: Mellanox Interconnect Community: Message List
Viewing all 6226 articles
Browse latest View live
↧

Re: it's normal speed?

October 21, 2015, 10:18 am
$
0
0

Hi Sophie,

1. OS - linux gentoo, i have 'ofed" in the repository (

sys-infiniband/ofed-3.12_rc1:3.12  USE="dapl diags ibacm mstflint opensm perftest qperf -rds -srp" OFED_DRIVERS="mlx4 -cxgb3 -cxgb4 (-ehca) -ipath -mlx5 -mthca -nes -ocrdma -psm" 0 KiB

 

2. Bios settings (C States - off, Hyper - Threading - enabled)

    OS tuning

 

#!/bin/sh

for cf in /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor;

do

echo performance > $cf;

done

 

echo 1000000000 > /proc/sys/kernel/shmmax

echo 400 > /proc/sys/vm/nr_hugepages

sysctl -w net.ipv4.tcp_timestamps=0

sysctl -w net.ipv4.tcp_sack=1

sysctl -w net.core.netdev_max_backlog=250000

sysctl -w net.core.rmem_max=4194304

sysctl -w net.core.wmem_max=4194304

sysctl -w net.core.rmem_default=4194304

sysctl -w net.core.wmem_default=4194304

sysctl -w net.core.optmem_max=4194304

sysctl -w net.ipv4.tcp_rmem="4096 87380 4194304"

sysctl -w net.ipv4.tcp_wmem="4096 65536 4194304"

sysctl -w net.ipv4.tcp_low_latency=1

 

NUMA - not support my CPU

 

3. All adaptersinstalledinPCIE GEN2x8

4. test speed

RDMA_Write BW Test

Dual-port       : OFF        Device         : mlx4_0

Number of qps   : 1        Transport type : IB

Connection type : RC        Using SRQ      : OFF

CQ Moderation   : 100

Mtu             : 4096[B]

Link type       : IB

Max inline data : 0[B]

rdma_cm QPs     : OFF

Data ex. method : Ethernet

---------------------------------------------------------------------------------------

---------------------------------------------------------------------------------------

#bytes     #iterations    BW peak[Gb/sec]    BW average[Gb/sec]   MsgRate[Mpps]

8388608    5000           13.72              13.72             0.000204

---------------------------------------------------------------------------------------

 

qperf 10.0.0.1 rc_bw

rc_bw:

    bw  =  1.71 GB/sec

 

It was better but there is no40gb

 

ps. Version ofed -

OFED_VER="3.12"

OFED_RC="1"

OFED_RC_VER="1"

OFED_SUFFIX="1.g8336567"

 

 

↧
Search

Re: How to dump mellanox connectx 3 libvma traffic with tcpdump?

October 22, 2015, 12:31 am
↧
↧

Read port priority counters is 0 .

October 22, 2015, 5:11 am
$
0
0

Hi.

 

I had a problem port priority  3 counters is not.

 

Reference is https://community.mellanox.com/docs/DOC-1414

 

What should we check

 

Thank you in advance!

 

 

 

pc1 - Switch - pc2

 

Switch firmware  3.4.2008

 

pc1 and 2 is Ububtu 14.04 

Fimware 2.35.5100

Driver 3.1.-1.0.3

 

 

pc1

 

root@Clotho:~# ib_write_bw -R --report_gbits --port=12500 -D 10

 

 

************************************

* Waiting for client to connect... *

************************************

---------------------------------------------------------------------------------------

                    RDMA_Write BW Test

Dual-port       : OFF          Device         : mlx4_0

Number of qps   : 1            Transport type : IB

Connection type : RC           Using SRQ      : OFF

CQ Moderation   : 100

Mtu             : 1024[B]

Link type       : Ethernet

Gid index       : 0

Max inline data : 0[B]

rdma_cm QPs     : ON

Data ex. method : rdma_cm

---------------------------------------------------------------------------------------

Waiting for client rdma_cm QP to connect

Please run the same command with the IB/RoCE interface IP

---------------------------------------------------------------------------------------

local address: LID 0000 QPN 0x025a PSN 0x86a3f8

GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:10:10

remote address: LID 0000 QPN 0x025a PSN 0x35cd7d

GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:10:20

---------------------------------------------------------------------------------------

#bytes     #iterations    BW peak[Gb/sec]    BW average[Gb/sec]   MsgRate[Mpps]

65536      417600           0.00               36.47              0.069568

---------------------------------------------------------------------------------------

 

root@Clotho:~# ethtool -S p1p1 | grep prio_3

     rx_pause_prio_3: 0

     rx_pause_duration_prio_3: 0

     rx_pause_transition_prio_3: 0

     tx_pause_prio_3: 0

     tx_pause_duration_prio_3: 0

     tx_pause_transition_prio_3: 0

     rx_prio_3_packets: 0

     rx_prio_3_bytes: 0

     tx_prio_3_packets: 0

     tx_prio_3_bytes: 0

 

root@Clotho:~# ./roce-check.sh

============================

  pfcrx / pfctx parameters 

============================

INTERFACE: mlx4_en

  RX: 0x08

  TX: 0x08

============================

  VLAN parameters          

============================

VLAN: p1p1.100

EGRESS priority mappings: 0:3 1:3 2:3 3:3 4:3 5:3 6:3 7:3

 

 

pc2.

 

ib_write_bw -R --report_gbits 192.168.10.10  --port=12500 -D 10

---------------------------------------------------------------------------------------

                    RDMA_Write BW Test

Dual-port       : OFF          Device         : mlx4_0

Number of qps   : 1            Transport type : IB

Connection type : RC           Using SRQ      : OFF

TX depth        : 128

CQ Moderation   : 100

Mtu             : 1024[B]

Link type       : Ethernet

Gid index       : 0

Max inline data : 0[B]

rdma_cm QPs     : ON

Data ex. method : rdma_cm

---------------------------------------------------------------------------------------

local address: LID 0000 QPN 0x025a PSN 0x35cd7d

GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:10:20

remote address: LID 0000 QPN 0x025a PSN 0x86a3f8

GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:10:10

---------------------------------------------------------------------------------------

#bytes     #iterations    BW peak[Gb/sec]    BW average[Gb/sec]   MsgRate[Mpps]

Conflicting CPU frequency values detected: 2799.937000 != 1220.531000. CPU Frequency is not max.

65536      417600           0.00               36.47              0.069568

---------------------------------------------------------------------------------------

 

 

root@Aprotos:~# ethtool -S p1p1 | grep prio_3

     rx_pause_prio_3: 0

     rx_pause_duration_prio_3: 0

     rx_pause_transition_prio_3: 0

     tx_pause_prio_3: 0

     tx_pause_duration_prio_3: 0

     tx_pause_transition_prio_3: 0

     rx_prio_3_packets: 0

     rx_prio_3_bytes: 0

     tx_prio_3_packets: 0

     tx_prio_3_bytes: 0

root@Aprotos:~# ./roce-check.sh

============================

  pfcrx / pfctx parameters 

============================

INTERFACE: mlx4_en

  RX: 0x08

  TX: 0x08

============================

  VLAN parameters          

============================

VLAN: p1p1.100

EGRESS priority mappings: 0:3 1:3 2:3 3:3 4:3 5:3 6:3 7:3

↧

Re: determine value of conflicts and fabric simulate

October 22, 2015, 7:37 am
$
0
0

Hi Serge.

 

1,2,3 - The short answer would be: while designing the topology you need to be sure to select the correct routing-algorithm. If selected correctly, it will prevent credit-loops and dead-locks.

There is a way to simulate a fabric and check the routing (to make sure there are no credit loops, etc...) however, it is a pretty complicated process that is done internally in Mellanox for large and non standard typologies.

Usually, we do it for complex projects as part of the Pre-Sales process.

I wrote you an email separately to initiate a more detailed discussion offline (In Russian ).

 

If I wanted to build a new cluster and invested a lot in buying the servers and the CPUs (most costly part of the infrastructure) I wouldn't try to "save" on networking as it may cause efficiency to be lower. Building a cluster and not utilizing it to the maximum (I mean CPU) doesn't make a lot of sense - don't you agree?

Reducing the number of links will cause most of the applications to provide lower performance so it is usually not recommended but possible.

 

Thanks,

Boris

↧

Re: SMB Direct

October 22, 2015, 9:19 am
↧
↧

Re: it's normal speed?

October 22, 2015, 10:34 am
↧

Re: How to switch ConnectX-3 from IB to ether mode on Windows Server

October 22, 2015, 2:18 pm
$
0
0

I resolved the issue. The MFT manual is wrong.

 

"SETTINGS_PORT1" is wrong.

"LINK_TYPE_P1" is the correct parameter name.

 

Also the value "3" (Auto) doesn't work in Windows, always recognized as IB and no "Port Protocol" tab showed.

So I had to set the value "2" (Ethernet) manually. After that my HCA was recognized as Ethernet card.

> mlxconfig -d mydevice set LINK_TYPE_P1=2 LINK_TYPE_P2=2

↧

Re: Mellanox ConnectX-3 SR-IOV problem

October 22, 2015, 3:15 pm
$
0
0

Hey, Kyle,

 

I am terribly sorry about the late reply.

 

After posting my original question and waiting for month, I wasn't able to resolve it. So I had to pause the work and disassemble my test bed.

 

I very much appreciate for your input, and will definitely use it as a helpful reference if I could ever resume the work. Really with Mellanox can offer more supports other than selling cards...

 

Best,

↧
Search

Re: SMB Direct

October 22, 2015, 10:24 pm
↧
↧

What is the latest (last?) firmware for MTS3600Q-1BNC and how do I start the sm?

October 23, 2015, 9:37 am
$
0
0

I recently acquired a 3600Q-1BNC. I'd like to upgrade it to the latest firmware. It isn't really clear from the download page which is the correct firmware, though I suspect it is: fw-IS4-rel-7_4_3000-MIS5025Q_A1-A3.bin.zip

 

Once it is running, I'd like to use the internal subnet manager rather than having to run opensm, but it says it needs a license. I thought an internal sm was provided with the BNC model? If so, how do I get the key and start it up?

 

Thanks!

Bob

↧

Re: determine value of conflicts and fabric simulate

October 23, 2015, 11:08 am
$
0
0

Thanks, Boris

 

Im anwer you an email.

↧

Re: Mellanox ConnectX-3 SR-IOV problem

October 25, 2015, 12:04 am
$
0
0

Dear Kyle,

 

how did you get the KVM guest to start at all? mine failed complaining that iommu group issue

 

root@vm-ha:~# virsh start kvm-node0

error: Failed to start domain kvm-node0

error: internal error: process exited while connecting to monitor: qemu-system-x86_64: -device vfio-pci,host=07:00.1,id=hostdev0,bus=pci.0,addr=0x6: vf

io: error, group 4 is not viable, please ensure all devices within the iommu_group are bound to their vfio bus driver.

qemu-system-x86_64: -device vfio-pci,host=07:00.1,id=hostdev0,bus=pci.0,addr=0x6: vfio: failed to get group 4

qemu-system-x86_64: -device vfio-pci,host=07:00.1,id=hostdev0,bus=pci.0,addr=0x6: Device initialization failed.

qemu-system-x86_64: -device vfio-pci,host=07:00.1,id=hostdev0,bus=pci.0,addr=0x6: Device 'vfio-pci' could not be initialized

 

root@vm-ha:~# lspci -s 07:00.1 -k

07:00.1 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3 Virtual Function]

        Subsystem: Mellanox Technologies Device 61b0

        Kernel driver in use: vfio-pci

root@vm-ha:~# lspci -s 07:00.2 -k

07:00.3 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3 Virtual Function]

        Subsystem: Mellanox Technologies Device 61b0

        Kernel driver in use: vfio-pci

root@vm-ha:~# lspci -s 07:00.0 -k

07:00.0 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3]

        Subsystem: Mellanox Technologies Device 0024

        Kernel driver in use: mlx4_core

 

I notice that my 07:00.0 and 07:00.1 device are both in the same iommmu group

 

I am also on Ubuntu 14.04.3 and Kernel 3.19

↧

Re: Connect-X adapter fails to receive invalid IPv6 frame

October 27, 2015, 11:19 am
$
0
0

If the packed doesn't pass a validation HCA is dropping it.

↧
↧

Re: Read port priority counters is 0 .

October 27, 2015, 11:37 am
$
0
0

Hi,

1.Can you send the output on the switch for the port priority counters?

2. Can you send the running config of the switch, to make sure PFC is enabled.

3. Which adapter is used?

4. can you send the output for this

RX=`cat /sys/module/mlx4_en/parameters/pfcrx`;printf "0x%x\n" $RX

and

TX=`cat /sys/module/mlx4_en/parameters/pfctx`;printf "0x%x\n" $TX

↧

Re: Is it possible to update the firmware on a Voltaire ISR9024D externally managed switched using a linux box?

October 28, 2015, 1:32 am
$
0
0

Hi Alfio,

 

Try to power off the switch, wait for a few minutes and then power it on again and try the below command:

 

spark -d **** -i verify

 

And then:

 

spark -d **** -i firmware.ISR9024D.img -nofs b

↧
Search

qperf version in RHEL6

October 28, 2015, 1:42 am
$
0
0

Trying to update a RHEL6 server with mlnx-ofed-all-3.0-1.0.1.noarch installed and having dependency issues.

When I run yum update all I get -

Error: Package: mlnx-ofed-all-3.0-1.0.1.noarch (@rh6-mellanox-x86_64)

           Requires: qperf = 0.4.9-1

           Removing: qperf-0.4.9-1.x86_64 (@rh6-mellanox-x86_64)

               qperf = 0.4.9-1

           Updated By: qperf-0.4.9-1.el6.x86_64 (x86_64-server-6_2015-10-01)

               qperf = 0.4.9-1.el6

 

So I can't install the same version!

Is the mellanox RPM hard coded to look for the exact qperf RPM rather than the version number?

If so, this needs fixing, if not I'll have to run a yum shell to sort this out.

 

Kernel version is 2.6.32-504.16.2.el6.x86_64

↧

Re: Connect-X adapter fails to receive invalid IPv6 frame

October 28, 2015, 2:10 am
$
0
0

In my use case, I need to use the adapter to send/receive invalid frames for testing purpose.

 

Do you know how to disable the validations you mention?

 

Thank you

↧
↧

Re: Are there any applications using RDS over InfiniBand?

October 28, 2015, 3:26 am
$
0
0

Pretty sure RDS is not tested nor actually supported (not to say it won't work) by Mellanox and the questions on it may better be directed to Oracle, including test matrix and API.

↧

Re: Are there any applications using RDS over InfiniBand?

October 28, 2015, 6:25 pm
$
0
0

Thanks Ferbs.

I found that RDS driver source has been integrated into MLNX_OFED, even though RDS is not tested and supported by Mellanox.

Besides, the driver code is always updated in the latest version of MLNX_OFED such as MLNX_OFED_LINUX-3.1-1.0.0-rhel6.5-x86_64, which is more newer than it in the latest version of Linux kernel-4.2.5/net/rds.

So what is the resource and motion to update RDS driver for MLNX_OFED?

Thanks for your help.

↧

Re: What is the latest (last?) firmware for MTS3600Q-1BNC and how do I start the sm?

October 29, 2015, 11:14 am
$
0
0

Hi Bob,

The 3600Q-1BNC (predecessor to the IS503X switches)  is listed on the EOL (end of life) page.The link below will let you download the latest SW image for this product.The SW image contains the FW and is automatically updated with the SW upgrade.To enable the SM you would need a FIT license if you have one the command to enable it is "ib smnode <your switch name> enable" example below.Be sure to do a "write memory" after changing any configuration so as to allow it to be persistent on reboot.

 

Example:

IS5030 [standalone: master] (config) # ib smnode IS5030 enable

 

Link to last GA version 1.1.2700 SW-

 

https://mellanox.my.salesforce.com/sfc/p/500000007heg/a/50000000PzrH/6mE4i9DhM3mvUm58KeDDYyfiVzmdxla1tPjBVU7CFgA

 

↧
Viewing all 6226 articles
Browse latest View live
Search
<script src="https://jsc.adskeeper.com/r/s/rssing.com.1596347.js" async> </script>