Quantcast
Channel: Mellanox Interconnect Community: Message List
Viewing all articles
Browse latest Browse all 6226

Re: MHGH28-XTC not working

$
0
0

No worries about the time gap.  We all get super busy at times and prioritise, etc.

 

With the problem you're experiencing, a fundamental bit of info is that OpenSM only attaches to one port when it runs.  By default, the first one it finds in a server (can be overridden in config file).

 

The way to think about it is that OpenSM starts up and locates the first Infiniband port, then explores/maps the network topology by finding whatever it can through that one port.

 

The reason I'm emphasising the "through that one port" bit, is to try and highlight that OpenSM won't see or recognise any of the other ports in that same server (unless there's an Infiniband switch in place to let the first port see the other ports).

 

One way to get around this is to have your all of your 3 nodes cabled from port 1 (on one box) to port 2 (on the next box), then run OpenSM on them all.  That way all ports will come up and be active.

 

I do this with a 2 node setup (port 1 of each box connecting to port 2 of the other, OpenSM running on both), then I run IPoIB over the top and set up individual IP subnetting for each group of ports so IP connectivity "just works".

 

I haven't yet tried it with a 3 node setup, but probably will do in a few weeks after I'm back in the UK.

 

Does this help?

 

(note - edited to improve clarity a bit)


Viewing all articles
Browse latest Browse all 6226

Trending Articles