Usually it can be originated from various reasons, I would start by making sure my servers are tuned accordingly from this document :
Are you using UD or CM on your IPoIB interfaces ? you might want to raise the buffer ring size and see if there any drops.
Also, in case the fabric is dirty, I would start by eliminating any misbehaving links. ibdiagnet is a nice tool to do that.
