Hi,
We have been trying to install DPDK-OVS on DL360 G7 (HP server) host using Fedora 21 and mellanox connectx-3 Pro NIC.
We used the several tutorials Gilad \ Olga have posted here and the installation seemed to be working up (including testpmd running - see output bellow).
We ran dpdk_nic_bind and didn’t see any user space driver we can bind to the mellanox device:
0000:06:00.0 'MT27520 Family [ConnectX-3 Pro]' if=ens1d1,ens1 drv=mlx4_core unused=ib_ipoib *Active*
1. We need to somehow bind this device to a DPDK-compatible driver, can you think of a way to do so ?
2. Can you please let take a look at the versions we use (fedora, OFED, dpdk, ovs, qemu) and let us know (from your experience) if we should upgrade\downgrade one of them ?
3. Do you have a more up to date tutorial for our specific HW?
4. Let us know if you need additional details.
Thanks a lot!!!!
===============
SYSTEM DETAILS:
===============
[root@localhost ~]# uname -a
Linux localhost.localdomain 3.17.4-301.fc21.x86_64 #1 SMP Thu Nov 27 19:09:10 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
Using Mellanox Technologies MT27520 Family [ConnectX-3 Pro] NIC
### Mellanox OFED version
MLNX_OFED_LINUX-3.1-1.0.3 (OFED-3.1-1.0.3):
OVS version: openvswitch-2.4.0
DPDK version: dpdk-2.1.0
QEMU version: qemu-2.2.1
##ethtool output
root@localhost ~]# ethtool -i ens1
driver: mlx4_en
version: 3.1-1.0.3 (29 Sep 2015)
firmware-version: 2.35.5100
bus-info: 0000:06:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: yes
root@localhost ~]# ethtool -i ens1
driver: mlx4_en
version: 3.1-1.0.3 (29 Sep 2015)
firmware-version: 2.35.5100
bus-info: 0000:06:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: yes
####################Test_PMD Output:
[root@localhost dpdk-2.1.0]# ./x86_64-ivshmem-linuxapp-gcc/build/app/test-pmd/testpmd -c 0xff00 -n 4 -w 0000:06:00.0 -- --rxq=2 --txq=2 -i
EAL: Detected lcore 0 as core 0 on socket 0
EAL: Detected lcore 1 as core 0 on socket 1
EAL: Detected lcore 2 as core 8 on socket 0
EAL: Detected lcore 3 as core 8 on socket 1
EAL: Detected lcore 4 as core 2 on socket 0
EAL: Detected lcore 5 as core 2 on socket 1
EAL: Detected lcore 6 as core 10 on socket 0
EAL: Detected lcore 7 as core 10 on socket 1
EAL: Detected lcore 8 as core 1 on socket 0
EAL: Detected lcore 9 as core 1 on socket 1
EAL: Detected lcore 10 as core 9 on socket 0
EAL: Detected lcore 11 as core 9 on socket 1
EAL: Detected lcore 12 as core 0 on socket 0
EAL: Detected lcore 13 as core 0 on socket 1
EAL: Detected lcore 14 as core 8 on socket 0
EAL: Detected lcore 15 as core 8 on socket 1
EAL: Detected lcore 16 as core 2 on socket 0
EAL: Detected lcore 17 as core 2 on socket 1
EAL: Detected lcore 18 as core 10 on socket 0
EAL: Detected lcore 19 as core 10 on socket 1
EAL: Detected lcore 20 as core 1 on socket 0
EAL: Detected lcore 21 as core 1 on socket 1
EAL: Detected lcore 22 as core 9 on socket 0
EAL: Detected lcore 23 as core 9 on socket 1
EAL: Support maximum 128 logical core(s) by configuration.
EAL: Detected 24 lcore(s)
EAL: VFIO modules not all loaded, skip VFIO support...
EAL: Searching for IVSHMEM devices...
EAL: No IVSHMEM configuration found!
EAL: Setting up physically contiguous memory...
EAL: Ask a virtual area of 0x200000000 bytes
EAL: Virtual area found at 0x7fa740000000 (size = 0x200000000)
EAL: Ask a virtual area of 0x200000000 bytes
EAL: Virtual area found at 0x7fa500000000 (size = 0x200000000)
EAL: Requesting 8 pages of size 1024MB from socket 0
EAL: Requesting 8 pages of size 1024MB from socket 1
EAL: TSC frequency is ~2666753 KHz
EAL: Master lcore 8 is ready (tid=c9c2e8c0;cpuset=[8])
EAL: lcore 14 is ready (tid=c58cd700;cpuset=[14])
EAL: lcore 12 is ready (tid=c68cf700;cpuset=[12])
EAL: lcore 13 is ready (tid=c60ce700;cpuset=[13])
EAL: lcore 10 is ready (tid=c78d1700;cpuset=[10])
EAL: lcore 15 is ready (tid=c50cc700;cpuset=[15])
EAL: lcore 11 is ready (tid=c70d0700;cpuset=[11])
EAL: lcore 9 is ready (tid=c80d2700;cpuset=[9])
EAL: PCI device 0000:06:00.0 on NUMA socket 0
EAL: probe driver: 15b3:1007 librte_pmd_mlx4
PMD: librte_pmd_mlx4: PCI information matches, using device "mlx4_0" (VF: false)
PMD: librte_pmd_mlx4: 2 port(s) detected
PMD: librte_pmd_mlx4: port 1 MAC address is e4:1d:2d:bb:6d:c0
PMD: librte_pmd_mlx4: port 2 MAC address is e4:1d:2d:bb:6d:c1
Interactive-mode selected
Configuring Port 0 (socket 0)
PMD: librte_pmd_mlx4: 0x20ad4740: TX queues number update: 0 -> 2
PMD: librte_pmd_mlx4: 0x20ad4740: RX queues number update: 0 -> 2
Port 0: E4:1D:2D:BB:6D:C0
Configuring Port 1 (socket 0)
PMD: librte_pmd_mlx4: 0x20ad5788: TX queues number update: 0 -> 2
PMD: librte_pmd_mlx4: 0x20ad5788: RX queues number update: 0 -> 2
Port 1: E4:1D:2D:BB:6D:C1
Checking link statuses...
Port 0 Link Up - speed 40000 Mbps - full-duplex
Port 1 Link Up - speed 40000 Mbps - full-duplex
Done
testpmd>
[root@localhost ~]# python /home/cloud/dpdk-2.1.0/tools/dpdk_nic_bind.py --status
Network devices using DPDK-compatible driver
============================================
<none>
Network devices using kernel driver
===================================
0000:03:00.0 'NetXtreme II BCM5709 Gigabit Ethernet' if=enp3s0f0 drv=bnx2 unused=ib_ipoib *Active*
0000:03:00.1 'NetXtreme II BCM5709 Gigabit Ethernet' if=enp3s0f1 drv=bnx2 unused=ib_ipoib
0000:04:00.0 'NetXtreme II BCM5709 Gigabit Ethernet' if=enp4s0f0 drv=bnx2 unused=ib_ipoib
0000:04:00.1 'NetXtreme II BCM5709 Gigabit Ethernet' if=enp4s0f1 drv=bnx2 unused=ib_ipoib
0000:06:00.0 'MT27520 Family [ConnectX-3 Pro]' if=ens1d1,ens1 drv=mlx4_core unused=ib_ipoib *Active*
Other network devices
=====================
<none>
========
TESTING:
========
We tried here to bind each of the available drivers to the device. However, none of them has caused the device to be using a DPDK-compatible driver.
python /home/cloud/dpdk-2.1.0/tools/dpdk_nic_bind.py --bind=ib_ipoib 0000:06:00.0
Routing table indicates that interface 0000:06:00.0 is active. Not modifying
[root@localhost cloud]# ifconfig ens1 down
[root@localhost cloud]# ifconfig ens1d1 down
[root@localhost cloud]# python /home/cloud/dpdk-2.1.0/tools/dpdk_nic_bind.py --bind=ib_ipoib 0000:06:00.0
Error: bind failed for 0000:06:00.0 - Cannot open /sys/bus/pci/drivers/ib_ipoib/new_id
//////////////////////////////////////////////////////////////////////////////////
[root@localhost cloud]# python /home/cloud/dpdk-2.1.0/tools/dpdk_nic_bind.py --bind=mlx4_core 0000:06:00.0
[root@localhost cloud]# python /home/cloud/dpdk-2.1.0/tools/dpdk_nic_bind.py --status
Network devices using DPDK-compatible driver
============================================
<none>
Network devices using kernel driver
===================================
0000:03:00.0 'NetXtreme II BCM5709 Gigabit Ethernet' if=enp3s0f0 drv=bnx2 unused=ib_ipoib *Active*
0000:03:00.1 'NetXtreme II BCM5709 Gigabit Ethernet' if=enp3s0f1 drv=bnx2 unused=ib_ipoib
0000:04:00.0 'NetXtreme II BCM5709 Gigabit Ethernet' if=enp4s0f0 drv=bnx2 unused=ib_ipoib
0000:04:00.1 'NetXtreme II BCM5709 Gigabit Ethernet' if=enp4s0f1 drv=bnx2 unused=ib_ipoib
0000:06:00.0 'MT27520 Family [ConnectX-3 Pro]' if=ens1d1,ens1 drv=mlx4_core unused=ib_ipoib *Active*
Other network devices
=================