> FC = Flow control, assuming it is enabled on the NIC (ethtool -a ethX) it is needed also to be enabled on every switch port on your Arista.
Got it. It is enabled on the NIC
$ ethtool -a p6p1
Pause parameters for p6p1:
Autonegotiate: off
RX: on
TX: on
But not enabled on the switch:
# show interfaces ethernet 14 flowcontrol
Port Send FlowControl Receive FlowControl RxPause TxPause
admin oper admin oper
--------- -------- -------- -------- -------- ------------- -------------
Et14 off off off off 0 0
Do the two devices need to agree on flow control before the NIC will try to pause? ethtool -S shows all the "rx_pause" counters are set to 0 (zero).
> Assuming you're using OFED 2.4 - Can you copy paste dmesg output when the low lever driver loads (mlx4_core) ?
So I have a variety of driver versions that seem to have this issue. I plan to get everything to 2.4 shortly. Here is the dmesg output from one server with 2.4:
$ dmesg | grep mlx
[ 17.650462] mlx4_core: Mellanox ConnectX core driver v2.4-1.0.4 (Apr 24 2015)
[ 17.737351] mlx4_core: Initializing 0000:02:00.0
[ 23.820087] mlx4_core 0000:02:00.0: PCIe link speed is 8.0GT/s, device supports 8.0GT/s
[ 23.908150] mlx4_core 0000:02:00.0: PCIe link width is x4, device supports x4
[ 26.069942] mlx4_core 0000:02:00.0: irq 86 for MSI/MSI-X
[ 26.069947] mlx4_core 0000:02:00.0: irq 87 for MSI/MSI-X
[ 26.069951] mlx4_core 0000:02:00.0: irq 88 for MSI/MSI-X
[ 26.069955] mlx4_core 0000:02:00.0: irq 89 for MSI/MSI-X
[ 26.069958] mlx4_core 0000:02:00.0: irq 90 for MSI/MSI-X
[ 26.069962] mlx4_core 0000:02:00.0: irq 91 for MSI/MSI-X
[ 26.069966] mlx4_core 0000:02:00.0: irq 92 for MSI/MSI-X
[ 26.069970] mlx4_core 0000:02:00.0: irq 93 for MSI/MSI-X
[ 26.069974] mlx4_core 0000:02:00.0: irq 94 for MSI/MSI-X
[ 26.069978] mlx4_core 0000:02:00.0: irq 95 for MSI/MSI-X
[ 26.069982] mlx4_core 0000:02:00.0: irq 96 for MSI/MSI-X
[ 26.069985] mlx4_core 0000:02:00.0: irq 97 for MSI/MSI-X
[ 26.069989] mlx4_core 0000:02:00.0: irq 98 for MSI/MSI-X
[ 26.114201] mlx4_en: Mellanox ConnectX HCA Ethernet driver v2.4-1.0.4 (Apr 24 2015)
[ 26.205113] mlx4_en 0000:02:00.0: registered PHC clock
[ 26.296333] mlx4_en 0000:02:00.0: Activating port:1
[ 26.402739] mlx4_en: eth0: Using 96 TX rings
[ 26.494296] mlx4_en: eth0: Using 8 RX rings
[ 26.586020] mlx4_en: eth0: Initializing port
[ 26.679114] <mlx4_ib> mlx4_ib_add: mlx4_ib: Mellanox ConnectX InfiniBand driver v2.4-1.0.4 (Apr 24 2015)
[ 26.775863] mlx4_core 0000:02:00.0: mlx4_ib_add: allocated counter index 1 for port 1
[ 28.825562] mlx4_en: p6p1: frag:0 - size:1522 prefix:0 stride:1536
[ 29.545220] mlx4_core 0000:02:00.0: mlx4_ib: Port 1 logical link is up
[ 29.545232] mlx4_en: p6p1: Link Up
Also:
$ ethtool --show-priv-flags p6p1
Private flags for p6p1:
pm_qos_request_low_latency : off
mlx4_rss_xor_hash_function : off
mlx4_flow_steering_ethernet_l2: on
mlx4_flow_steering_ipv4 : on
mlx4_flow_steering_tcp : on
mlx4_flow_steering_udp : on
qcn_disable_32_14_4_e : off
blueflame : on
> and you can also try optimizing the flow steering mode - just create a mlnx.conf file under /etc/modprobe.d/ and put the following line :
>
> options mlx4_core log_num_mgm_entry_size=-7
Seems like for most use cases, this would be a good thing to do. Any reason it isn't the default? I'll give it a try. Thanks again for your help. -- Bud