Trying to configure eIPoIB so that a guest VM running under KVM on a compute node can access a storage node on the Infiniband network.
Both the compute host and storage node are running Ubuntu 12.04 with the Mellanox OFED 2.1 with IPoIB loaded.
Using iperf with the storage node as server and compute node as client, iperf gives me about 15 Gb/s.
When I load eIPoIB on the compute host and run iperf from the guest as the client (storage node as sever), I get .13 Gb/s. While I don't know what performance I should expect, that seems a little off!
I think I have something misconfigured between ib0, br100 and eth2. Can anyone help me with a sample /etc/network/interfaces file that works?
As far as I can tell, eIPoIB got loaded and is operating properly on the compute host:
root@compute4:/sys/class/net# cat /sys/class/net/eth_ipoib_interfaces
eth2 over IB port: ib0
root@compute4:/sys/class/net# cat /sys/class/net/eth2/eth/*
SLAVE=ib0.1 EMAC=01:00:5e:00:00:fb IMAC=00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:00:00:00:fb
SLAVE=ib0.2 MAC=52:54:00:a5:f7:79 IP=172.23.0.201 VLAN=N/A
ib0.1
ib0.2
SLAVE=ib0.1 MAC=00:25:90:17:61:09 VLAN=N/A
SLAVE=ib0.2 MAC=52:54:00:a5:f7:79 VLAN=N/A
root@compute4:/sys/class/net# brctl show
bridge name bridge id STP enabled interfaces
br100 8000.002590176109 no eth2
vnet1
virbr0 8000.fe54000833c9 yes
root@compute4:/sys/class/net# ifconfig br100
br100 Link encap:Ethernet HWaddr 00:25:90:17:61:09
inet addr:172.23.0.202 Bcast:172.23.0.255 Mask:255.255.255.0
inet6 addr: fe80::225:90ff:fe17:6109/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:2044 Metric:1
RX packets:66 errors:0 dropped:0 overruns:0 frame:0
TX packets:52 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:2538 (2.5 KB) TX bytes:9621 (9.6 KB)
vnet0