I've read the Linux Kernel IPoIB and NFS-RDMA documentation. It doesn't seem particularly current with the 2.6 kernels mentioned.
I'm looking for documentations for doing NFS-RDMA with the linux kernel 3.13 (which is used by Ubuntu 14.04 LTS) and mellanox FDR cards and switch.
I've got RDMA and IPoIB working. So ping, ibv_srq_pingping, udaddy, rdma_server, ib_send_bw, rping, and ucmatose all show client <-> server connectivity.
Even the mount works:
root@c2-31:/# mount -o rdma,port=20049 10.18.3.34:/export/1 /mnt/ib
root@c2-31:/#
I can cd around, but whenever I try to actually read a file I get:
root@c2-31:/mnt/ib/bill# ls -alh testfile
-rw-r--r-- 1 root root 1.0G Feb 20 19:31 testfile
root@c2-31:/mnt/ib/bill# cat testfile
cat: testfile: Input/output error
On both client and server I do have:
# ulimit -l
unlimited
# tail -4 /etc/security/limits.conf
* soft memlock unlimited
* hard memlock unlimited
root soft memlock unlimited
root hard memlock unlimited
My goal for this is to get it working and write a puppet module to manage the client and server setup for NFS RDMA.