I am using the ConnectX-3 EN (BCBT) in a Centos 6.5 server to generate packets (proprietary packet generation software) for my test network. Most of the time, the NIC operates as expected but over the course of my 12 hour test run, I see the NIC get into a state where every 256th packet is sent on the wire tagged with VLAN 0. When in this state, wireshark on the Linux server shows that the outgoing packet has my configured VLAN tag (VLAN id = 11) but if I Wireshark the receiving port, I see every 256th packet has the wrong VLAN id (0). This leads me to believe it is a problem in the driver/firmware (since wireshark is reporting the packet as sent to the NIC for tx). Once in the offending state, I connected two ConnectX-3 EN (BCBT) back to back (to rule out my test network) and I still see the issue.
Other notes:
- I see the a VLAN 0 tag get inserted even if I am transmitting untagged packets
- The problem is transient - disappears for a while, then comes back
- I have the latest firmware and driver (details attached)
- I see the problem with or without tx_vlan option enabled (managed with ethtool)
- I reproduced the problem on 2 Mellanox ConnectX-3 EN (BCBT) cards so it is not specific to one defective device.
- I am using the NICs in promiscuous mode and not assigning a VLAN to the NIC with ifconfig
Is this a known issue? Has anyone seen anything similar?
Many thanks,
Mike
Attached are the details of the packet as transmitted and received, the ethool output for the NIC in question, and server details.