Quantcast
Channel: Mellanox Interconnect Community: Message List
Viewing all 6226 articles
Browse latest View live

R vs Python

$
0
0

When it comes to selecting a preferable data analysis language, I believe most of you will agree with me that R and Python should be talked about at the same time. It is very hard to pick one out of these two amazingly flexible data analytics languages.

 

I can confess that I have yet to pick the best out of these two data scientist delights. Therefore, to make things interesting, I will be presenting some curated information about these two languages and will leave the decision-making in the hands of my readers. It is worth mentioning that there are multiple resources available to understand the strengths and weaknesses of both the languages. However, in my opinion, there is a strong relationship between these two languages.

Stack Overflow Trends

Image title

The above graph shows how these two languages have trended over time based on the use of their tags since 2008, when Stack Overflow was founded.

While both R and Python are competing to be the data scientist’s language of choice, let’s look at their platform share and compare 2016 with 2017

Image title

Now, it is time to look at these two languages from the perspective of their usage, data handling capabilities, task and ease of installing, and getting started with them.

 

Usage

R is used the most when data analysis tasks require standalone computing or individual servers. Python is a glue language, therefore, it is generally used when data analysis tasks require integration with web applications or when a piece of statistical code needs to be inserted into a production database.

 

Tasks

R wins hands down when it comes to performing exploratory statistical analyses. It is considered to be easy for beginners. Statistical models can be written with few lines of code. Python as a full-fledged programming language can be a great tool to deploy algorithms for production use.

 

Data Handling Capabilities

R is handy when it comes to a multitude of packages for both coders and non-coders to not only perform statistical tests but also to create machine learning models.

Python has had its own challenges related to data analysis. However, after the introduction of NumPy, Pandas, and a few others, it has started gaining a lot of popularity in the field of data analytics, as well.

 

How to Get Started

For R, you need IDE R Studio. For Python, there are many Python IDEs to pick from. However, the Spyder and IPython Notebook are the most popular ones.

 

Popular Packages and Libraries

Let's look at popular packages and lbraries for R and for Python, for coders and non-coders alike.

R: Popular Packages for Coders

  • dplyr, plyr, and data table for data manipulation
  • stringr to manipulate strings
  • zoo to work with regular and irregular time series
  • ggvis, lattice, and ggplot2 data visualization
  • caret for machine learning

 

R: Popular Packages for Non-Coders

  • Rattle
  • R Commander
  • Deducer

These are full-blown GUI packages that can help in performing amazing statistical and model creation routines.

 

Python: Popular Libraries for Coders

  • pandas for data manipulation
  • SciPy/NumPy for scientific computing
  • scikit-learn for machine learning
  • matplotlib for graphics
  • statsmodels to explore data, estimate statistical models, and perform statistical tests and unit tests

 

Python: Popular Libraries for Non-Coders

Orange Canvas 3.0 is an open-source software package released under GPL. It uses common Python open-source libraries for scientific computing such as numpy, scipy, and scikit-learn.

 

R and Python Trivia

R language

Python language

Creator

Ross Ihaka and Robert Gentleman

Guido van Rossum

Release date

1995

1991
Must-knows
  • R is an implementation of S language (Bell Labs).
  • R’s design and evolution are handled by R-core group and R foundation.
  • R’s software environment was written in C, Fortran, and R.
  • Python was inspired by C, Modula-3, and (in particular) ABC.
  • Python gets its name from the “Monty Python’s Flying Circus” comedy series.
  • Python Software Foundation (PSF) takes care of Python’s advances.
PurposeR is focused on user-friendly data analysis, statistics, and graphical models.

Python emphasizes productivity and code readability.

Usability
  • Statistical model can be written with only a few lines.
  • There are R stylesheets but not everyone uses them.
  • The same piece of functionality can be written in different ways.
  • Nice Syntax enables easier coding and debugging within Python.
  • The code indentation can affect its meaning.
  • Any piece of functionality is always written in the same way.
Ease of learning
  • R has a steep learning curve at the start. However, once you know the basics, you can easily learn the complex stuff.
  • R is not hard for experienced programmers.
  • Python’s readability and simplicity make its learning curve relatively low and gradual.
  • Python is considered to be a good language for starting programmers.

 

As I mentioned at the start of this article, there is a strong tie-in between R and Python, and both of these languages are gaining popularity day-by-day. And to make it even harder to pick which one is better, the integration of these two languages has caused a lot of positive and collaborative ripples within the data science community.

 

Conclusion

Day-to-day users and data scientists are getting best of both worlds, as R users can run a rPython package within R to run Python code from R, and Python users who are using RPy2 library can run R code from within the Python environment.


Re: Can't ping IPV6

$
0
0

Hi David,

 

What is the mlnx-os version running on tbe switches?
"Show version"

Re: Can't ping IPV6

Re: Can't ping IPV6

$
0
0

Hi David,

 

Please open a support ticket for this at support@mellanox.com

we have a known issue with 3.6.4112 on SwitchX devices and ipv6 - we will provide you with a workaround for this version.

the permanent fix will be available in MLNX-OS 3.6.5000 due in start of November

Re: Disabling Spanning tree breaks the network?

$
0
0

Hi David,

 

If you didn't configure MLAG on the 2 Mellanox switches - you don't need to disable STP.

Nutanix recommendation is related when MLAG is configured.

Re: Disabling Spanning tree breaks the network?

$
0
0

Understood! Is there any benefit of deploying mlag compared to lacp For the switch interconnection?

Re: Problema con Tarjeta infiniband Qlogic 4x QDR IB PCI-E G2 HCA

$
0
0

If you'll have a look at the HO Qlogic 4x QDR IB PCI-e G2 HCA overview (see: https://support.hpe.com/hpsc/doc/public/display?docId=emr_na-c01951526 ) then you'll find that this card is based on the Mellanox Connectx-2 IB technology, in which unfortunately is not supported by Mellanox WinOF-driver v5.35 (supports only ConnectX-3 / Connect-X 3 Pro )  - see: http://www.mellanox.com/related-docs/prod_software/MLNX_VPI_WinOF_Release_Notes_Rev_5.35.pdf

WinOF driver v4.80 is the last Mellanox driver that supports ConnectX-2 based adapters over Windows 2012 R2. Can be retrieved from Mellaox website

In case adapter is still not revealed properly by “device-manager” then suggesting to apply to HP/Qlogic and ask to advice you if & what driver & FW they can supply to that adapter.

Generally speaking – Mellanox already declared Connectx-2 adapters as EOL

Re: Unstable ConnectX-3 Ethernet Performance on ESXi 6.5 update 1

$
0
0

Would suggest that you start first with:

1. “Performance Tuning” for Mellanox Adapters & driver, to ensure you have proper BIOS configuration, to check the cpu core frequency and that you use PCIe generation that suit the adapter etc…

https://community.mellanox.com/docs/DOC-2489#jive_content_id_Getting_started

2. Check the Mellanox inbox driver by using "esxcli software vib list |grep nmlx4" command.

for example:
# esxcli software vib list |grep nmlx4
Name                           Version                               Vendor  Acceptance Level  Install Date
-----------------------------  ------------------------------------  ------  ----------------  ------------
nmlx4-core                     3.0.0.0-1vmw.600.0.0.2494585          VMware  VMwareCertified   2015-12-16
nmlx4-en                       3.0.0.0-1vmw.600.0.0.2494585          VMware  VMwareCertified   2015-12-16
nmlx4-rdma                     3.0.0.0-1vmw.600.0.0.2494585          VMware  VMwareCertified   2015-12-16

3. Assuming that the56Gb nic is up check the netqueue is enabled by default.
netqueue.jpg

4. Set nmlx4_en module parameters num_rings_per_rss_queue to 4.
# esxcli system module parameters set -m nmlx4_en -p "num_rings_per_rss_queue=4"
# reboot
# esxcli system module parameters list -m nmlx4_en

In case the suggested above is implemented and you still have low performance, then apply to Mellanox support (support@mellanox.com) to get further assistance on this


Re: Unstable ConnectX-3 Ethernet Performance on ESXi 6.5 update 1

$
0
0

Hi!

Performance was more lower then before num_rings_per_rss_queue=4 option.

 

I can't find netq option in inbox driver & I can't access your link netqueue.jpg.

 

Is there another solution?

 

Ethernet link-up speed is industry standard between every vendor.

It's a serious problem.

 

If your HCA achieve maximum speed on RDMA only, just say it that our product focus RDMA communication only.

 

Your every product brief said support 40, 50, 56, 100, 200Gb Ethernet.

Why should I open then support ticket for your problem?

 

Why always your staff click a correct answer?

I think who request the question must click a correct answer button.

 

BR,

Jae-Hoon Choi

Re: SX6018 to an Connect-IB MCB191A gives me only 40Gbps, not 56Gbps

$
0
0

Hi Joakim,

 

Can you provide the part number of the switch?

output of "show inventory" is fine.

or via the web

 

System -> Modules

or

System -> Inventory

 

I suspect the part number is SX6018T which means "SwitchX®-2 based 18-port QSFP FDR10 1U managed InfiniBand switch system"

Re: SFP+ NIC mnpa19-xtr Vmware

$
0
0

Generally speaking, these is no Mellanox driver available that supports ConnectX-2 over Esxi6.0

However, the latest Mellanox driver that supports ConnectX-2 adapter is mlnx_ofed_esxi v1.9.10.0, over Esxi5.5 with fw v2.9.1200 and above.

See in Mellanox web-site: http://www.mellanox.com/page/products_dyn?product_family=29

I doubt whether VMware have any inbox driver that may supports ConnectX-2 EN adapter over Esxi6.0 but you can try to get they advice on this as well

Notice that Mellanox has already declared ConnectX-2 EN MNPA19-XTR network adapter as EOL (end-of-life) therefore Mellanox is not providing any support related to this card.

Scalability issue for multiple clients

$
0
0

Hello,

 

Our setup is

1 x Mellanox MX354A Dual port FDR CX3 adapter w/1 x QSA adapter

1 x Xeon E5-2450 processor (8 cores, 2.1Ghz)

16GB Memory (4 x 2GB RDIMMs, 1.6Ghz)

 

We have 4-node cluster, and all of them are server and client at the same time.

When write, a node split data into 4 pieces and concurrently write to 4 nodes.

When read, a node read from 4 nodes.

 

We expect it scales with the number of multiple clients.

when a node is reading, it can get 6.4 GB/s bandwidth

but when 2 nodes are reading, both only get 5GB/s each, although aggregated bandwidth is enough.

 

There's only 1 CPU, no NUMA discrepancy arises.

Concerned possible NIC cache misses, measured PCIe Read using pcm-pcie.

Simply PCIe read cannot scale with increasing number of clients even if its bandwidth is actually much higher.

 

There must be contention when multiple connections(QPs) read from a single server.

Please Mellanox, can you pinpoint the root cause and possible solution for multiple-client scalability?

Re: SX6018 to an Connect-IB MCB191A gives me only 40Gbps, not 56Gbps

$
0
0

Part number indeed starts with , so it seems you're right. Is this actually a completely different switch, or is the 56Gbps full FDR speed a software option?

Re: SX6018 to an Connect-IB MCB191A gives me only 40Gbps, not 56Gbps

$
0
0

Hi Joakim,

 

it's the same ASIC but it's restricted by software and fw. we can't bypass this.
also the price is differnet between the FDR10 and FDR modules

Re: Unstable ConnectX-3 Ethernet Performance on ESXi 6.5 update 1

$
0
0

Hi!

I find this link about vmxnet3 vNIC performance limitation.

 

Network Improvements in vSphere 6 Boost Performance for 40G NICs - VMware VROOM! Blog - VMware Blogs

 

I think this limitation leads to performance problems.

 

I'm also change to Global Pause mode on SX6036G that shows me a solid slightly under 10Gbps performance level.

 

Is there another solution?

 

BR,

Jae-Hoon Choi

 

P.S

 

This screenshot is my POC infrastructures.

pNIC lists.JPG

 

When I query about rdma device list to my system, it shows me a 10 Gbps speed on ConnectX3.

Is it correct?


Re: Error MessageSystem.QueryException: unexpected token: '('

$
0
0

Hi,

 

Sorry for the dumb question: can you please specify which SW package/product this exception is coming from? Version of it?

Please describe the scenario on which you received this exception.

 

...

Re: disable a port and you will not be able to bring it back up

$
0
0

Hi Leje,

 

The Mellanox adapter ConnectX VPI is already EOF (end-of-life) in terms of sale & Mellanox support. So I suggest you overview the advanced ConnectX-3/4/5 adapters that will provide you the required throughput.

 

Thanks

mlx5 direct verbs example

$
0
0

Is there any end-to-end example application code for mlx5 direct verbs? I want to use the strided RQ feature.

 

The best that I have found until now is the mlx5 transport for UCX, which implements functionality similar to mlx5dv. But it's part of a bigger codebase and not cleanly separable.

Mellanox ConnectX-5 adapter speed 650MB/s only

$
0
0

Helllo,

I have two Mellanox Connectx-5 adapters connected back to back. Link speed shows 40Gbps. I am using widows server 2016. Tried with different PCIe slots and tested with iperf3. But in case of TCP speed reaches up to 650 MB/s only and in UDP it only achieves nearby 450 MB/s.

I am sure I am missing out some setting but can not figure out what.

Can anybody help me in this case?

HowTo Uninstall ONIE/Cumulus and Replace it with MLNX-OS

$
0
0

Hi everyone,

 

I'am in phase to buy 4 x SN2100-CB2FC, so delivered with a preload Cumulus.

  - 3 of them will be mainly used for training / testing / labs with my students and professionnal learner.

  - The last one will be, perhaps, used in my own stable environnement.  (Yes ! 100G are awesome to play with !)

 

My questions are:

 

1/ I would have to regularly reset three of them in order to go back to a ''clean'' environnement.

     I have read many things about it, but apparently i will need to ''reactivate'' the Cumulus license each time i reset. Is it right ? If so, is it always the same key or do i need to call Mellanox each time ?

 

2/ On the one that will be used for myself. I would like to uninstall ONIE/Cumulus and replace it with MLNX-OS. Cause i am mainly familiared with it.

     Is it possible ? In my opinion, reading this article : HowTo Uninstall MLNX-OS and Install ONIE on Mellanox Switches , it looks...

     Do I need to purchase a licence of MLNX-OS ? Or will it be integrated (''offered'') with the switch, as this is a Ethernet switch ?

 

Many thanks for all your responses !

Sincerely yours !

Viewing all 6226 articles
Browse latest View live


<script src="https://jsc.adskeeper.com/r/s/rssing.com.1596347.js" async> </script>