OK, I've been trying to setup eIPoIB. I have my infiniband network up, ib0 is setup for ipoib I can see the new eth2 interface (virtual eIPoIB device) but nothing in the vifs, so I can't ping from compute node to compute node.
Welcome to Ubuntu 12.04.2 LTS (GNU/Linux 3.2.0-39-generic x86_64) * Documentation: https://help.ubuntu.com/ System information as of Thu May 9 22:31:11 EST 2013 System load: 1.83 Users logged in: 0 Usage of /: 2.7% of 59.39GB IP address for eth0: 192.168.10.101 Memory usage: 0% IP address for ib0: 10.10.10.101 Swap usage: 0% IP address for eth2: 20.20.20.101 Processes: 112 Graph this data and manage this system at https://landscape.canonical.com/ Last login: Thu May 9 21:55:22 2013 from maas.local ubuntu@blade01:~$ sudo su - root@blade01:~# cat /sys/class/net/eth2/eth/vifs root@blade01:~# root@blade01:~# root@blade01:~# ibstat CA 'mlx4_0' CA type: MT25418 Number of ports: 2 Firmware version: 2.8.0 Hardware version: a0 Node GUID: 0x001b78ffff33ee58 System image GUID: 0x001b78ffff33ee5b Port 1: State: Active Physical state: LinkUp Rate: 20 Base lid: 5 LMC: 0 SM lid: 3 Capability mask: 0x02510868 Port GUID: 0x001b78ffff33ee59 Link layer: InfiniBand Port 2: State: Active Physical state: LinkUp Rate: 20 Base lid: 2 LMC: 0 SM lid: 1 Capability mask: 0x02510868 Port GUID: 0x001b78ffff33ee5a Link layer: InfiniBand root@blade01:~# ifconfig ib0 ib0 Link encap:UNSPEC HWaddr A0-00-01-00-FE-80-00-00-00-00-00-00-00-00-00-00 inet addr:10.10.10.101 Bcast:10.10.10.255 Mask:255.255.255.0 inet6 addr: fe80::21b:78ff:ff33:ee59/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:65520 Metric:1 RX packets:6 errors:0 dropped:0 overruns:0 frame:0 TX packets:30 errors:0 dropped:8 overruns:0 carrier:0 collisions:0 txqueuelen:1024 RX bytes:1515 (1.5 KB) TX bytes:5752 (5.7 KB) root@blade01:~# ping 10.10.10.102 PING 10.10.10.102 (10.10.10.102) 56(84) bytes of data. 64 bytes from 10.10.10.102: icmp_req=2 ttl=64 time=2.24 ms 64 bytes from 10.10.10.102: icmp_req=3 ttl=64 time=0.033 ms ^C --- 10.10.10.102 ping statistics --- 3 packets transmitted, 2 received, 33% packet loss, time 2000ms rtt min/avg/max/mdev = 0.033/1.141/2.249/1.108 ms root@blade01:~# ifconfig eth2 eth2 Link encap:Ethernet HWaddr 00:1b:78:33:ee:59 inet addr:20.20.20.101 Bcast:20.20.20.255 Mask:255.255.255.0 inet6 addr: fe80::21b:78ff:fe33:ee59/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) root@blade01:~# ping 20.20.20.102 PING 20.20.20.102 (20.20.20.102) 56(84) bytes of data. ^C --- 20.20.20.102 ping statistics --- 3 packets transmitted, 0 received, 100% packet loss, time 2015ms root@blade01:~# cat /sys/class/net/eth2/eth/vifs root@blade01:~# ???? root@blade01:~# ethtool -i eth2 driver: eth_ipoib version: 1.0.0 firmware-version: 1 bus-info: ib0 supports-statistics: yes supports-test: no supports-eeprom-access: no supports-register-dump: no root@blade01:~#