Hi,
1. I suggest you start with the MLNX_OFED or MLNX_EN User Manual located on Mellanox web.
Mellanox Products: Mellanox OpenFabrics Enterprise Distribution for Linux (MLNX_OFED)
2. I suggest you also browse here Solutions
You may find many thing related to mlnx_qos, for example:
Network Considerations for Global Pause, PFC and QoS with Mellanox Switches and Adapters
End-to-End QoS Configuration for Mellanox Switches and Adapters
HowTo Configure QoS over SR-IOV
There are more discussion with answer that may help you.
Ophir.