Forgot your password?
typodupeerror
Python Open Source

Is Python a Legitimate Data Analysis Tool? 67

Posted by Soulskill
from the one-end-it-is dept.
Back in May we discussed using Python, R, and Octave as data analysis tools, and compared the relative strength of each. One point of contention was whether Python could be considered a legitimate tool for such work. Now, Bei Lu writes while Python on its own may be lacking, Python with packages is very much up to the task: "My passion with Python started with its natural language processing capability when paired with the Natural Language Toolkit (NLTK). Considering the growing need for text mining to extract content themes and reader sentiments (just to name a few functions), I believe Python+packages will serve as more mainstream analytical tools beyond the academic arena." She also discusses an emerging set of solutions for R which let it better handle big data.
This discussion has been archived. No new comments can be posted.

Is Python a Legitimate Data Analysis Tool?

Comments Filter:
  • really? (Score:2, Interesting)

    by Anonymous Coward

    Any Turing-complete language is a legitimate data analysis tool.

    • by Meshach (578918)

      Any Turing-complete language is a legitimate data analysis tool.

      The question is not whether or not it is possible but whether or not it is realistic and practical.

      • by KiloByte (825081)

        With the right libraries, it ALWAYS is both realistic and practical.

        Of course, you'd need really good libraries to overcome malbolge or brainfuck, but hey, no one says the underlying language has to be visible from behind them...

      • by jonadab (583620)
        > The question is not whether or not it is possible but whether or not it is realistic and practical.

        Using Python for data analysis is realistic, assuming you know Python (or have enough background in computer science to pick it up quickly -- it's not a particularly difficult language, as languages go: I've seen accounting software packages that would be much harder to learn).

        Python is perhaps not quite as practical as some other choices. In particular, object-oriented programming is not an especially
      • The question is not whether or not it is possible but whether or not it is realistic and practical.

        Not only is it realistic and practical but it is already in use for data analysis! Everyone on the ATLAS experiment at CERN uses python to some degree in their analysis and my grad students and I use an analysis framework almost entirely in Python with ROOT [root.cern.ch] for I/O.

    • Any Turing-complete language is a legitimate data analysis tool.

      Legitimately =/= feasible without regards to cost.

      Otherwise, let's use assembly to write our own analytics package.</rollseyes>

  • It Works (Score:5, Insightful)

    by mrsquid0 (1335303) on Friday July 06, 2012 @01:56PM (#40567633) Homepage

    Python may not be a legitimate data analysis tool, but it is widely used for data analysis, and it gives the right results. For the most part that is what really matters.

    • Re:It Works (Score:5, Insightful)

      by mcgrew (92797) * on Friday July 06, 2012 @02:01PM (#40567699) Homepage Journal

      Python is a language. It's a tool to build other tools with, including data analysis tools.

      • by Instine (963303)
        or use other libraries easily and quickly. PyCUDA gives genuinely huge number crunching power to the language. And allows meta programming which suits scripting languages and machine learning very well. http://mathema.tician.de/dl/pub/nvidia-gtc-2009.mp4 [tician.de]

        The readability and flexibility and speed of development are what it brings, the raw power comes from the libraries it can talk to.
    • Re:It Works (Score:5, Insightful)

      by ceoyoyo (59147) on Friday July 06, 2012 @02:19PM (#40568005)

      What does "legitimate data analysis tool" mean? MatLab was included in the comparison, and MatLab is more of an engineering tool. The built in (excuse me, optional paid for) stats library is pretty limited.

      R is great for doing statistical analysis, but it's not great for doing things like image analysis. Without additional libraries R isn't nearly as good as it is with libraries either.

      • Re:It Works (Score:4, Funny)

        by roman_mir (125474) on Friday July 06, 2012 @07:06PM (#40571419) Homepage Journal

        What does "legitimate data analysis tool" mean?

        - obviously it means to ask whether Python is legitimate or is bastard, what do you think it means? It is not asking whether Python is a 'data analysis tool', it is asking whether Python is a legitimate something or other.

        So to answer the question you have to look at the Python's descendancy. You'll quickly discover that Python was actually conceived in a huge orgy of different programming paradigms, styles and languages, it's even named after a circus!

        I believe the answer is that Python is a bastard of data analysis tools, but so what, bastards are people too.

  • by Anonymous Coward

    Someone who knows so little about tools like R, python, etc. should spend their time learning about what is available rather than writing articles on the topic using their own cursory knowledge.

  • Use what works (Score:5, Insightful)

    by hawguy (1600213) on Friday July 06, 2012 @02:12PM (#40567889)

    Since people do use python for data analysis (hence the data analysis related packages that are available), of course it's legitimate.

    Just like how when you're standing on the roof and you need to pound in a couple nails, that heavy pair of pliers in your pocket is a legitimate tool. It may not be the best tool for the job, the best tool might be a pneumatic nail gun, but if all you have with you and what you know how to use is pliers, then that's the right tool. Why spend time and money learning some other "more appropriate" language (or buying an air compressor and nail gun) when you already have a tool at your fingertips that will do what you need.

    As your needs grow you might need to find another more appropriate tool, but if you can get the job done with Python, why bother searching for the "perfect" tool?

    Depending on your needs, sh, awk, sed, sort, and uniq may be all the tools you need - many log parsing, analysis and reporting programs have been writing with those tools, often ingesting more rows of data per day than many small business BI systems.

    • Why spend time and money learning some other "more appropriate" language (or buying an air compressor and nail gun) when you already have a tool at your fingertips that will do what you need.

      Indeed, although sometimes you save yourself a lot of headaches by getting a tool that was built for your task. I have, in a pinch, used a screw driver to hammer nails, but a screw driver is no replacement for a hammer.

      That being said, Python+SciPy+NumPy is fine for data analysis; people use it all the time, and it works as well as R or MatLab. It is not as though we are talking about QuickBasic for data analysis.

  • by Anonymous Coward on Friday July 06, 2012 @02:20PM (#40568025)

    http://xkcd.com/353/

  • by Anonymous Coward

    I looked at R and it's one of the most deranged languages I've ever seen in terms of syntax (up there with Erlang). At least Python is readable to the average programmer who knows C or Java.

  • by Anonymous Coward

    I work in the biosciences and we occasionally have a similiar discussion.

    In our context, it isn't about how one analyzes the the data, it is a question about how anyone else can recreate your experiment: that is, set up the experimental system, acquire the data, analyze it which will yield approximately the same results. It is in our best interest [and mandated by our funding agency and the journals] to publish papers that clearly define how we made our observations and how we analyzed the data.

    My group con

  • "legitimate" is such a disrespectful value judgment. Are you saying that people who do data analysis with Python are illegitimate? Are you calling them bastards?

    No, seriously, you can have a profitable conversation all about the reasons why you think there are serious drawbacks to using Python as your data analysis tool. Lots of people might benefit from that. But when you start saying things like "That's not a legitimate data analysis tool" or "That's not a real programming language" or whatever, then

    • by Sir_Sri (199544)

      legitimate" is such a disrespectful value judgment. Are you saying that people who do data analysis with Python are illegitimate? Are you calling them bastards?

      I'm not sure that's how it's meant, but I agree, it's an odd choice of phrase. If I were to look at it another way, what would make a language 'illegitimate' for data analysis? In that case you look at things like excel and access for financial transactions, or some of the early versions of CUDA that didn't support proper IEEE floating point maths (or at least, not fast IEEE floating point maths). In those cases you can use the language, and it will spit out results, but they might not be right, and ther

  • by Anonymous Coward

    Just ask the astronomy community. They've been moving away from IDL as an analysis environment and towards the use of python with scipy (with numpy and pyfits offering similar performance). You're asking this question several years after it's already been effectively declared as such.

  • This story seems like an echo of the one a day or so ago about Linux being critical to the success of the LHC. Something with generic programmability supports something specific, then gets discussed as a tool for that specific task. Probably a lot of the comments there apply here.
  • by gizmo_mathboy (43426) on Friday July 06, 2012 @07:28PM (#40571615)

    Python and Perl make great data analysis tools.

    They have a plethora libraries to handle things: Numpy/Scipy for Python and PDL/GSL for Perl.

    They can access FORTRAN and C libraries as necessary for either performance or legacy needs.

    THey are probably best because they are high level languages, very platform neutral, and cost signficantly less than other "serious" data analysis tools/languages.

  • by Anonymous Coward

    Yes absolutely. Its being used to do all sort of data analysis in the real world.

    Check out Pandas (http://pandas.pydata.org/) the Python data analysis library.

    Also there are lots of machine learning libraries: scikits-learn is probably the best known (http://scikit-learn.org/)
    Both of these are built on NumPy.

    You should also check out the videos from the 2012 PyData workshop: http://marakana.com/s/2012_pydata_workshop,1090/index.html

  • I love R and Python. However, both of them choke on big data sets. What they need is an in-built mechanism to store data on disk rather than in-memory. There are some really convoluted ways of doing this..but then dont always work with modeling packages that weren't written with the convoluted approach you are taking, in mind. So, if the base language has the ability to store object on disk, say with a simple flag, and its transparent to the rest of the system, most downstream libraries/packages would still

The reason that every major university maintains a department of mathematics is that it's cheaper than institutionalizing all those people.

Working...