I have had this issue before. Using atomics on Connect-IB requires different queue pair setup flags than ConnectX-3 (for example, IBV_EXP_QP_CREATE_ATOMIC_BE_REPLY), and probably requires using "experimental" verbs. mlx4-based atomics code DOES NOT work with mlx5. The perftest library contains an example of this setup, which I was able to modify for my application.
Anuj