I'm currently working a task to migrate some software from RHEL 6.3 to RHEL 6.7 (no, 7 isn't an option). I'm having issues getting our RDMA connection online, on our Connect-X2 cards. Specifically, we get an RDMA_CM_EVENT_CONNECT_ERROR. The only configuration difference I've been able to track down relates to the MTU settings.
In the working 6.3, the Interface showed IB0 as having an MTU of 2044, and ibv_devinfo showed IB0 to have max_mtu and active_mtu of 2048.
In 6.7, the interface still shows IB0 using MTU 2044, but ibv_devinfo shows IB0 to have a max_mtu and active_mtu of 4096.
I did try using set_4k_mtu=0 in /etc/modprob.d/mlx4_core.conf, with no effect.