Thanks for the reply.
I was misinformed by folks that I spoke to earlier that 100 Gb could be achieved on the ConnectIB device transparently to an application. This is apparently posssible for IPoIB applications that use an ethernet bonding capablility but not native verbs applications. (I was assured that it was not just an ethernet thing but also a verbs thing...)
The sales brochures on the website show the nice S curves achieving 100Gb/s but do not explain how that was achieved unfortunately. This was the documentation I was referring to.
In my tests on a 16X slot in an HP server I was able to see 6.3 GB/s on the two links simultaneously as expected. This was achieved with little advanced configuration needed. So everything is good, just a little confusion.
We look forward to the connect IB product and connectX 4. These days you can never have enough IO!