Hi Nils,
Turns out that you need to add this "-x MXM_OOB_FIRST_SL=0" to your mpirun on your cluster.
Otherwise if you do a pstack on a process which it is hung, you would find that the process is hung in some routine in hcoll, because hcoll used the pml for OOB messaging.
Anyway, this seems to work works for me:
$ mpirun -np 2 -host nxt0111,nxt0110 -x MXM_OOB_FIRST_SL=0 ${HPCX_MPI_TESTS_DIR}/imb/IMB-MPI1