I have a strange issue. I've set ib mode on FreeBSD driver 2.1.6 (same behavior on 2.1.5), and set ip for ib0. The issue that i'm facing is that after a reboot, i need to do ifconfig ib0 down && ifconfig ib0 up to actually have link on ib0. Also i must do this after "mlx4_core0: mlx4_ib: Port 1 logical link is up".
See here:
#dmesg ... mlx4_core0: <mlx4_core> mem 0xfb200000-0xfb2fffff,0xf9000000-0xf97fffff irq 40 at device 0.0 on pci4 uhub0: 2 ports with 2 removable, self powered uhub1: 2 ports with 2 removable, self powered ugen1.2: <vendor 0x8087> at usbus1 uhub2: <vendor 0x8087 product 0x0024, class 9/0, rev 2.00/0.00, addr 2> on usbus1 ugen0.2: <vendor 0x8087> at usbus0 uhub3: <vendor 0x8087 product 0x0024, class 9/0, rev 2.00/0.00, addr 2> on usbus0 uhub3: 6 ports with 6 removable, self powered uhub2: 8 ports with 8 removable, self powered ugen0.3: <Winbond Electronics Corp> at usbus0 ukbd0: <Winbond Electronics Corp Hermon USB hidmouse Device, class 0/0, rev 1.10/0.01, addr 3> on usbus0 kbd0 at ukbd0 mlxen0: Ethernet address: e4:1d:2d:0d:bf:60 mlx4_en: mlx4_core0: Port 1: Using 12 TX rings mlx4_en: mlx4_core0: Port 1: Using 8 RX rings mlx4_en: mlxen0: Using 12 TX rings mlx4_en: mlxen0: Using 8 RX rings mlx4_en: mlxen0: Initializing port mlx4_core0: mlx4_ib_add: allocated counter index 1 for port 1 SMP: AP CPU #1 Launched! SMP: AP CPU #4 Launched! SMP: AP CPU #6 Launched! SMP: AP CPU #9 Launched! SMP: AP CPU #7 Launched! SMP: AP CPU #8 Launched! SMP: AP CPU #5 Launched! SMP: AP CPU #11 Launched! SMP: AP CPU #10 Launched! SMP: AP CPU #3 Launched! SMP: AP CPU #2 Launched! Timecounter "TSC" frequency 2000046460 Hz quality 1000 Trying to mount root from zfs:zroot/ROOT/default []... ib0: Attached to mlx4_0 port 1 ums0: <Winbond Electronics Corp Hermon USB hidmouse Device, class 0/0, rev 1.10/0.01, addr 3> on usbus0 ums0: 3 buttons and [Z] coordinates ID=0 igb0: link state changed to UP lagg0: link state changed to UP vlan200: link state changed to UP igb1: link state changed to UP igb2: link state changed to UP igb3: link state changed to UP mlx4_core0: mlx4_ib: Port 1 logical link is up root@fat:~ # ifconfig ib0 ib0: flags=8043<UP,BROADCAST,RUNNING,MULTICAST> metric 0 mtu 4092 options=8009b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,LINKSTATE> lladdr 0.0.0.48.fe.80.0.0.0.0.0.0.e4.1d.2d.3.0.d.bf.61 inet 10.105.0.230 netmask 0xffffff00 broadcast 10.105.0.255 nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> root@fat:~ # ping 10.105.0.225 PING 10.105.0.225 (10.105.0.225): 56 data bytes ping: sendto: No route to host ping: sendto: No route to host ^C --- 10.105.0.225 ping statistics --- 2 packets transmitted, 0 packets received, 100.0% packet loss root@fat:~ # ifconfig ib0 down && ifconfig ib0 up root@fat:~ # ping 10.105.0.225 PING 10.105.0.225 (10.105.0.225): 56 data bytes 64 bytes from 10.105.0.225: icmp_seq=0 ttl=64 time=0.208 ms ^C --- 10.105.0.225 ping statistics --- 1 packets transmitted, 1 packets received, 0.0% packet loss round-trip min/avg/max/stddev = 0.208/0.208/0.208/0.000 ms root@fat:~ # ifconfig ib0 ib0: flags=8043<UP,BROADCAST,RUNNING,MULTICAST> metric 0 mtu 4092 options=8009b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,LINKSTATE> lladdr 0.0.0.48.fe.80.0.0.0.0.0.0.e4.1d.2d.3.0.d.bf.61 inet 10.105.0.230 netmask 0xffffff00 broadcast 10.105.0.255 nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
What i have in config:
/boot/loader.conf mlx4_core="YES" ibcore_load="YES" mlxen_load="YES" mlx4ib_load="YES" ipoib_load="YES" /etc/sysctl.conf sys.device.mlx4_core0.mlx4_port1=ib /etc/rc.conf ifconfig_ib0="inet 10.105.0.230 netmask 255.255.255.0 up mtu 4092"