OK...so i've made some more progress.
A. I can get connections with the linux hosts with all cables, with IP addresses assigned i can ping other computers over the IB fabric. Still can't get IBping to work, I think i'm using wrong GUID or something. I also found some of my issues were due to a faulty card.
B. Subnet manager is working. Previous issues with subnet manager disappeared after a reboot, possibly related to fault card from A
C. My connectx card was upgraded to a new firmware, will try it today to at least get two machines on fabric. Main issue now seems to be the fact I bought mellanox infinihost III cards, which apparently aren't supported anymore under server 2012. I've got new connect-x cards coming that I will hopefully get better luck with. So now i'm dead in the water until those arrive.