Dear Yun,
I was able to install
I downloaded ubuntu 14.10 package from mlnx site and installed on my ubuntu 14.04.
IPoIB, iSER and iSCSI all work flawlessly. However, I cannot get ib_srpt to work under 2.4-1.0.4. Interesting thing is that I was able to make kernel 3.19.5 and ib_srpt work under MLNX_OFED Driver 2.2-1.
My system setup are the following.
OS: ubuntu 14.04
Kernel: linux-headers-3.19.5-031905-generic
MLNX_OFED Driver: 2.4-1.0.4
and the setup works on ipoib, srp, iscsi & iSER. However, I am not able to get ib_srpt to work with mlnx_ofed 2.4 (but able to make it work with ubuntu distro ofed 2.2-1)
The trick is download the latest supported 3.19 kernel from ubuntu main ppa: Index of /~kernel-ppa/mainline/v3.19.5-vivid
And download the latest mlnx_ofed 2.4 driver for ubuntu 14.10, after extra the files, change distro file from ubuntu 14.10 to ubuntu 14.04.
Then you can run the mlnxofedinstall
1. Which version of driver did you install?MLNX-OFED-ESX-1.9.10.0-10EM-550.0.0.1331820 What's your HCA?CX354 P/N: MCX354A-FCCT
2. What target are you using? I'm testing your partner storage company: Israel zadara storage.Don't know the target to use what type
2. What target are you using? If iser target with TGT?
1.9.10 is the Ethernet driver it does not support SRP it only support iSER.
1.8.3 is the IB driver which support both iSER and SRP.
That's why you are not seeing SRP in vSphere. I am not familiar with Zadra. But looks like it's ZFS based. If they do support IB you could load 1.8.3 IB driver to your ESXi to see if there is any performance difference.
*I do not work for Mellanox but I do like their product
I am trying to use Connectx-3EN and Connectx-pro adapters in SRIOV mode inside a PowerKVM host. Issue what I am facing is realted to VF-VF communication on the same NIC port/PF. I had created 4 VFs on each ofthe cards what I had. Then created 2 guests [RHEL 71.LE] and assigned 1 VF each to them via pci passthrough. Assigned IP addressed both of VFs [same subnet used], but I am not able to communicate between two guests using the VF interfaces.. Is there any additional configuration which I need to do? I had referred to below doc on your website, which has a comment at last saying 'Yes, the card support local bridging ( loopback). bridging is on the same e-switch ( currently same port) .'
https://community.mellanox.com/docs/DOC-1123
But for me its not working. I looked for any utilities which Mellanox has to interact with the eswitch, but found only eswitch module for openstack which cannot be used for my purpose. Appreciate any pointers or help here.
Thank you.
Srikanth Aithal
IBM
Hi, Chris
Thanks for the info, we've also investigated a little, there are plenty of compat code in 2.4.0 tarball for
version <=3.17, so seems like the project is based on 3.18+.
While there are many issues we found when trying to merge the project into our own tree, but anyway,
I think it could support 3.18 at least :-)
Hi,
To the best of my knowledge, NVGRE has no offload support by the Linux Kernel. Only Windows OS is supported.
Your best option right now is to use VxLAN.
Thanks
Since this can go many directions, I suggest you open a ticket with Mellanox support. let the folks debug your issue.
send your issue to support@mellanox.com
This one is very well priced and - depending on your location - the postage is decent too:
http://www.ebay.co.uk/itm/301613347569
Note - I don't have any relationship to the seller, I just typed in "Voltaire ISR 9024D-M" on eBay.
Hmmm, see if you can get a hold of a Voltaire ISR 9024D-M. They're probably the best 20Gb/s speed combination of capabilities & noise. Anything that's SDR is likely to be extremely noisy, so you'll want to avoid them.
That being said, I haven't used any of the Flextronics switches in person. So that one may be an exception. There's more info about IB switches suitable for home use here, if it helps:Suggestions for *quiet* Infiniband switch?
Oh, those are ancient Infiniband cards. Very little modern software will work with them.
For RHEL 6 though you may still be in luck. If you install the "Infiniband Support" group through yum (which comes with RHEL 6), that should find the cards. eg:
$ sudo yum groupinstall "InfiniBand Support"
Got around to doing this finally. .stl files in case anyone else finds them useful:
I had 45 cards. So far of the 45, I have moved 20 systems to RHEL 6, 18 have worked. The Mellanox software I tried couldn't undate the firmware, but at least the RHEL 6 made most of them visible.
Which software have you tried for burning the firmware, and what version?
From memory (it's been a long time), you'll need to use "flint" prior to the 3.x.x releases. eg 2.7.2 should work:
http://www.mellanox.com/downloads/MFT/mstflint_x64_2_7_2.zip
Other flint versions should be downloadable here, if I chose the wrong one for that link: Mellanox Technologies
It may be better to use the verion of flint (mstflint?) that's packages in RHEL 6. I don't have a box handy to check it on right atm.
Guessing you've figured out how to retrieve the model number and revision of the cards? It's in the output of lspci above. Be sure to double check each card individually. I've had different revision numbers of the same card happen in the same batch (needed different firmware). No idea if that's common though.
I have 2x SX1036 switches,
Each node has one VPI PRO dual port adapter, connected to both switches,
At this moment I am using mellanox for all my networking, except for a small dedicated 1g copper network exclusively for the management-network. (IPMI etc.)
If possible I would like to avoid copper where possible, since I am trying to minimize my latency, especially my broadband/WAN latency. (used to connect to Internet)
My ISP placed a "FTU" in my house, this is where the fiber is terminated, and the ISP network stops, customer network begins. The box contains of 2 parts; the ftu and ntu. In the NTU the fiber gets converted to copper , and the NTU spits out a cat6 cable with 1g copper connection. I was told I can connect it to any router as long as it can do vlan's and PPPoE. The thing is; I want to lose the whole NTU thing, and just avoid the converting to copper. I think I could try to disconnect the 2 ISP fibers from the ftu and attach both fibers to a SFP adapter/plug (btw, could I directly use a SFP+ adapter for this ? That will save the converting SFP to SFP+, I'm guessing this would otherwise add latency?
There is also a possibility that I can find a NTU that does no converting but has sfp/sfp+ cable as output. This would be less stressful for the ISP Fibers, since they are connected to the FTU/NTU, and from there a new SFP cable does all the dirty work. I was wondering if anyone could give my an estimate on how much latency this would add compared to directly to sfp+? (or SFP, please explain if SFP+ directly is possible)
Could you guys give some advice regarding above situation/questions?
Now lets get to the biggest problem so far;
I Cant find PPPoE in my Mellanox SX1036 !!! I am beginning to think that this switch/router does not support PPPoE. What would be the best way to go about adding PPPoE support to the SX1036? And if entirely not possible, what would be the best/fastest way to add it to the network? Using a router in front of the SX1036's means extra hop, and probably big $$$ for sfp+ router? What do you guys think is the best way to go about this ?
I really appreciate your help!!
Cloudbuilder
Hi Bent Vader,
I will try to answer some of your questions:
Q1 So, This gets us to my first question; What do you guys think about this idea? Please give me your thoughts, and tell me if I'm missing something huge here.
My second question is about cabling, and there effect on performance (mostly latency/jitter, i want my connection to have lowest possible latency and jitter, Since my remote users would benefit from a crystal clear link. to my ISP
Q2 So, I'm checking out providers, and most 500/500 or 1000/1000 providers install a GeneXis OCG-1012 or similar modems/fiber termination points. Some even install there own router also, and both of these are of course RJ45 based. So, How would I go about connecting this RJ45 port to one of my SX1036 ports, while adding the most minimum amount of latency/delay or other issues to the connection.
A2: you can use the Mellanox RJ45 to SPF+ trasceiver combined the QSA (QSFP to SFP adapter):
From:
Cables - Mellanox Technologies
QSFP-SFP+ module - MAM1Q00A-QSA
RJ45-SFP+ module - MC3208411-T
Q3 I Understand that I will need to split QSFP first, would u advice on split 1 to 4 or 1 to 1 in this case? (any difference as far as performance/stability?)
A3: I would advice not splitting but just using a regular QSFP mode and changing it's speed to 1000Mbps (1Gbps).
I learned that each router/switch etc. in a network adds most latency, so these should be avoided right? Is there a way to convert a RJ45 cable to something that goes in a SFP+ port? Or would this be dumb. because of the added copper wire for the RJ45 cable, and should we pick a way that uses fiber all the way to the RJ45 port ? Does this kind of equipment exist? How much would the benefit be going with this route?
Q4 I think even better (perhaps the best, read next q ) than that would be to find a provider that will put a custom modem with SFP+ port at my site, so I can direct-connect the SX1036 to the modem without any converters except for the splitter from QSFP to SFP+, but i doubt i'll find one that will. Most modems that do i know of have SFP ports anyway, not SFP... so I would still need to convert that to SFP+ right?
Q5 What is your opinion on all the solutions discussed so far, will it even matter? would it shave off 1ms during a ping using the best solution compared with the Modem->extra router->switch with SFP+ port->Mellanox switch solution? shave off 0.1? Are there any other things as latency that would benefit from a better solution? Jitter? packetloss ? Im not that well-educated in network performance after BW and latency.
What would you do in my case? Go for the simple solution or the optimal solution since it will actually make a difference? (remember we talking doing 3d VDI over the internet, and webhosting that needs to be as fast as possible )
Q6Logic tells me that the converting with extra routers and switches/converters should be avoided as the plague, but I'm wondering 2 things; in the end, would choices like these really make a difference, or are we talking about such small numbers that it doesn't really matter at all?
Q7What would the absolute best solution be in terms of performance/latency ? Is it possible to terminate the fibers at the SX1036? Thereby leaving the whole modem out of the picture? Has anyone heard of an ISP that allows such a thing? prop. not. So lets say the fiber-end unit HAS to be placed by the ISP, what would be best in this case? Direct SFP+ port in the ISP placed Modem to QSFP splitter? Or QSFP port in modem, QSFP cable to SX1036 directly? Would the difference be these 2 be even measurable in terms of ping? 0.01ms? or even far less than that.
Questions about hardware / way of setting this up.
Q8 Would there be anymore advice you want to share on running my network this way? Are there any other people doing this? Maybe some Pro/Cons?
Q9 Would 2 SX1036 with L3 and 56gbe even be enough in terms of switches/gear to setup my network like this, in a redundant way? (I have far less nodes than ports ofc.)
Q10 What protocol should I set up? OSPF MLAG+whatever? What would my options be, and what are the pro's/con's for each?
Q 11 Most nodes as of now have one card with dual ports, each port goes to one of the switches, how would I best cable the switches ? Should they even be connected?
Q 12 Is there any problem mixing connectx-3 VPI, connectx-3 VPI PRO and connectx-3 EN and connectx-3 EN PRO cards together in one setup? What card would you guys advice on and why?
I'll be using GPU's in some hypervisors, if I want to use GPU-Direct between nodes that would only be possible if both nodes have a connect-X3 VPI PRO right? This is not possible on the non-pro VPI nor on either of the EN cards correct?
Hi Bent,
The MLNX-OS software doesn't support PPPoE.
I would suggest opening a ticket at support@mellanox.com to open a feature a request so this feature would be considered for a future release.
Hi,
I could not find this answered in the discussion group.
As far as I can tell the Mellanox HPC-X MPI is built without the option: --enable-mpi-thread-multiple
to the OpenMPI configure script. Is this intentional, and if so: why?
To test and verify try the osu benchmark sample that comes with HPC-X:
$ mpirun -host localhost -x MLX5_SINGLE_THREADED=n -np 2 $HPCX_MPI_TESTS_DIR/osu-micro-benchmarks-4.4.1/osu_latency_mt
[1430755920.480363] [login0401:2589 :0] sys.c:741 MXM WARN Conflicting CPU frequencies detected, using: 2601.00
[1430755920.479771] [login0401:2590 :0] sys.c:741 MXM WARN Conflicting CPU frequencies detected, using: 2601.00
MPI_Init_thread must return MPI_THREAD_MULTIPLE!
-------------------------------------------------------
Primary job terminated normally, but 1 process returned a non-zero exit code.. Per user-direction, the job has been aborted.
...
This test never runs due to the MPI run-time not providing the sufficient level of threads support.
/Nils
Hello,
Where i can find default firmware for SX6018 switch?
Both images appears to be corrupt.
Stuck on u-boot... 2 images at FF400000 and FF600000 but both with problems.
I am using an HP c7000 enclosure with 4 BL685c G1 blades, and a HP 4x DDR IB Swith Module. The blades are running Vmware ESXi 5.1.0 U2 . I am using the Mellanox ofed stack ( MLNX-OFED-ESX-1.8.1.0 ) and Mellanox driver mlx4_en-mlnx-1.6.1.2 . The addapters are recognized by the Vmware as follows :
# esxcli network nic list | grep 10G
vmnic_ib0 0000:047:00.0 ib_ipoib Up 20000 Full 00:23:7d:94:d8:7d 2044 Mellanox Technologies MT25418 [ConnectX VPI - 10GigE / IB DDR, PCIe 2.0 2.5GT/s]
vmnic_ib1 0000:047:00.0 ib_ipoib Up 20000 Full 00:23:7d:94:d8:7e 2044 Mellanox Technologies MT25418 [ConnectX VPI - 10GigE / IB DDR, PCIe 2.0 2.5GT/s]
I am also using a Topsin 120 switch with Subnet Manager running on it, the MTU in the vSwitch and portgroup is set to 2044, the switch is capable of handling 2048. So MTU-wise all is ok.
And the thing that botheres me is that when measuring the speed between 2 nodes with iperf I get speeds around 4-5 Gbits/sec . And to be honest with 4xddr I was expecting something in the ballpark of 7-8-9 Gbits/sec .
Is there something that I am doing wrong ?
Any input will be appreciated.
Thanks in advance.