I was recently engaging with Mellanox technical support about the vNIC limitations on BX5020 gateways in various modes of operation, and was told:
The "vnics-mode 4k" limitation allows up to 4,000 total vNics on the gateway.
For this to work correctly, the "single-gw-mode" must also be enabled in the group. With these settings configured, there should be no problems increasing your vNic count above 1024.
If you weren't including Ethernet ports in the port channel, then you could actually set the "vnics-mode" to "unlimited".
But, in your configuration 4K will be the limitation.
This response gave rise to some follow-up questions that I haven't been able to get Mellanox to answer, so I thought I'd ask the community in case anyone knows or has experience with this.
Our current configuration is that we have a 2x10GbE LAG (port-channel) bridged to an InfiniBand fabric with around 950 hosts on it. The reason for my initial question is that we are adding to the fabric and will soon be exceeding 1,000 nodes (and 1,000 vNICs) and I wanted to make sure we wouldn't run into a 1k VNIC limit. It sounds like we will not, but now I am wondering the following:
1) What is meant by "not including Ethernet ports in the port channel" in order to be able to set unlimitied-VNICs mode? What other ports could you include in a port channel on a BX5020? Doesn't the BX5020 consist of only Ethernet and InfiniBand ports (which don't really have a concept of port channels, as the subnet manager handles the parallel routes)? This response makes me wonder if I have some sort of fundamental misunderstanding of the BX5020 as a system.
2) Other than allowing one to increase the vNIC limit to 4k from 1k, what are the effects of turning on single-gw-mode? What does single-gw-mode do? I have not been able to find an answer to this in the documentation.
Thanks in advance for any answers or suggestions,
Rusty D