Hi Fergus-
Can you tell me a little more about your application and existing network infrastructure? What bandwidth are you using currently, and are you planning on increasing it? How many hosts are there? This is Ethernet, right? What are your current bottlenecks?
If CPU usage and latency are your top concerns, moving to RDMA is definitely where you will see the big wins. Moving to RoCE (RDMA over converged Ethernet) will fully offload data transfers from the CPU to the NIC, bypassing the OS networking stack, which gives large improvements for CPU usage, latency, and overall bandwidth. Network Direct (ND) is the API in Windows for RDMA, but it requires modifications for your application. If your application does (or can) use SMB, the necessary support is built into Windows and you get it for free. Keep in mind your fabric (switches) have to support RDMA and flow control as well.
Here are some articles on RDMA that may be useful: http://www.mellanox.com/blog/tag/rdma/
--Walker.
Walker Hagius
Solutions Engineer
Mellanox Technologies