I would like to suggest to start with latest software/firmware and follow instructions in Mellanox Tuning Guide Mellanox Products: Mellanox OpenFabrics Enterprise Distribution for Linux (MLNX_OFED)
Common mistakes are usually that system is not tuned, or/and application is running on wrong CPU