Forgot your password?
typodupeerror
Programming Stats Math Open Source Python

R Throwdown Challenge 185

Posted by timothy
from the if-you-pirate-it-so-much-the-better dept.
theodp (442580) writes "'R beats Python!' screams the headline at Prof. Norm Matloff's Mad (Data) Scientist blog. 'R beats Julia! Anyone else wanna challenge R?' Not that he has anything against Python, Matloff adds, but he just doesn't believe that Python or Julia will become 'the new R' anytime soon, or ever. Why? 'R is written by statisticians, for statisticians,' explains Matloff. 'It matters. An Argentinian chef, say, who wants to make Japanese sushi may get all the ingredients right, but likely it just won't work out quite the same. Similarly, a Pythonista could certainly cook up some code for some statistical procedure by reading a statistics book, but it wouldn't be quite same. It would likely be missing some things of interest to the practicing statistician. And R is Statistically Correct.'"
This discussion has been archived. No new comments can be posted.

R Throwdown Challenge

Comments Filter:
  • by smitty_one_each (243267) * on Sunday May 25, 2014 @07:30AM (#47086779) Homepage Journal
    Nothing with a name that verbose can possibly be any good.
  • And R is Statistically Correct.

    I don't see any margin of error. This claim is scientifically worthless.

  • Bad analogy (Score:5, Insightful)

    by Florian Weimer (88405) <fw@deneb.enyo.de> on Sunday May 25, 2014 @07:39AM (#47086797) Homepage

    An Argentinian chef is more likely to make great sushi than a Japanese automotive engineer.

    You generally want to use programming languages designed by experienced programmers (even better, experienced language designers) who work closely with subject matter experts. Left to their own devices, experts are likely to get a lot of things wrong, and if the language is sufficiently popular, you are stuck with their mistakes for a long time to come.

    • Re:Bad analogy (Score:5, Interesting)

      by Glock27 (446276) on Sunday May 25, 2014 @08:08AM (#47086871)

      Exactly. Julia will eat R for lunch soon enough, I think. It's an elegant, well designed and efficient language. It's only been around for a couple of years, and has a very vibrant and rapidly growing community.

      Check it out for yourself: The Julia Language Homepage [julialang.org]. It's got a lot to offer anyone with an interest in mathematics, including statisticians. It's based on the LLVM, and interfaces trivially with C libraries - plus it's a very fast language in it's own right, unlike R or Python.

      • Re:Bad analogy (Score:5, Interesting)

        by retchdog (1319261) on Sunday May 25, 2014 @08:48AM (#47086959) Journal

        my friend uses julia, and every few weeks complains about some bug. the other day he mentioned that the latest release broke Bernoulli sampling (wtf?). the others have been pretty fundamental too.

        this is a serious problem, of course. the other one is lack of libraries. R is an abysmal pile of shit, but at least it's a standard; pretty much 95%+ of applied stats is at least partially supported by someone's hacked-up library/package. julia is far, far short of that, and it appears that much of its community is more interested in pretty graphics, meta-wankery, and interface methodology than actual working statistics (not that there's anything wrong with that per se).

        yeah, yeah, "fix it yourself," and it's on my list to write at least a basic survival analysis package for it. but i wouldn't blame anyone for not using it, and i wouldn't recommend it for doing stats as it is now.

        • by Gaygirlie (1657131) <gaygirlie@hotmaiLIONl.com minus cat> on Sunday May 25, 2014 @08:59AM (#47086995) Homepage

          my friend uses julia, and every few weeks complains about some bug.

          He should tell Julia to wear protection and be more careful with who she spends time with so as not to catch so many bugs.

        • How much R package code is written in R? Would it be such a problem to take an R parser and generate Julia code out of it as a first iteration? Then, people could refactor it - if necessary - while keeping the first version around for regression testing. Even if the original R APIs are horrible, at least they have the benefit of people being familiar with them, as you rightly point out.
          • Like the f2c toolkit, for converting Fortran to C?

            I don't think you could write he parser in R, or in Julia.

            • Or f2cl? ;-) I don't see a reason why one shouldn't be able to write the parser in Julia. It seems perfectly equipped even for such tasks. It even has macros, come to think of it.
          • Given that these languages are (primarily, obviously anything Turing-complete can be turned to the same purposes as anything else, if somebody feels like it) used for statistics work, I'd be inclined to wonder whether that is the easiest or best way to go about it:

            If something is already implemented in R, and you want to more or less blindly feed it a new target, or re-run it to see how it works, R was apparently not broken enough to stop it, because it's already done.

            If you want to implement some, cu
            • Julia isn't strictly numerical. It sure as hell isn't "primarily for statistics work". It has a numerical bent, but so far I haven't seen any limitation in the sense that something general and non-numeric in it would be possible (in the sense of Turing completeness) but impractical. Indeed, the very fact that Julia has been designed with support for Lisp-like macros in mind should be a hint to you that perhaps expecting it to have at least generous facilities for manipulating and transforming syntactic tree
          • Although much R package code is written in R, many of the important bits are living in FORTRAN libraries (many of which date back to the 1980s) which are linked into the packages.

      • by HiThere (15173)

        Julia is an excellent design for a specific range of problems. I was considering using it for a couple of days, so I looked over the design. It is good for handling matricies of identical types of element doing the same thing on each entry. This is a pretty broad class of problem, but it's far from descriptive of all problems, and even within that class I'm skeptical that they will ever be able to optimise some of the operations.

        OTOH, I must admit that I didn't even consider using R. I wasn't considerin

        • > It is good for handling matricies of identical types of element doing the same thing on each entry.

          Actually, that's MATLAB. Julia does not give matrices any special treatment - it has a type system that is rich enough that you can define an entire matrix domain-specific-language inside it (which is exactly what they did - Julia's matrix operations are defined entirely in Julia itself, yet are still blazing fast because they call external libraries). http://julia.readthedocs.org/e... [readthedocs.org]

          Plus, whereas MATLA

      • How can a non-functional language be _the_ platform for mathematical computing?

    • Using three lines of code I can do a regression in R and get the output, including loading the data.

      Python? Fuhgeddaboutit. Can do, but with a lot more code.

      Of course, if you're looking to do stuff you'd expect of a normal scripting language, R falls flat on its face.

      The solution? R + Python. They talk to each other quite nicely, and you can get the best of both worlds.

      • by tomhath (637240)

        Python? Fuhgeddaboutit. Can do, but with a lot more code.

        Yea, with Python it takes up to nine lines of code [blogspot.com] to calculate the regression and generate a plot

        • Re:Bad analogy (Score:4, Informative)

          by KingOfBLASH (620432) on Sunday May 25, 2014 @09:29AM (#47087107) Journal

          You're just getting a plot. I'm talking about output that looks like this:


          Call:
          lm(formula = new_day_return ~ prior_day_return + rsi_under_10 +
                  rsi_under_20 + rsi_under_30 + rsi_over_70 + rsi_over_80 +
                  rsi_over_90 + fourteen_day_rsi, data = mydata5)

          Residuals:
                Min 1Q Median 3Q Max
              -100 -1 0 1 205700

          Coefficients:
                                                Estimate Std. Error t value Pr(>|t|)
          (Intercept) -9.845e+01 3.742e+02 -0.263 0.792
          prior_day_return -4.143e-04 3.434e-03 -0.121 0.904
          rsi_under_10 -1.916e-01 3.798e+00 -0.050 0.960
          rsi_under_20 2.195e-02 1.447e+00 0.015 0.988
          rsi_under_30 -2.291e-01 6.915e-01 -0.331 0.740
          rsi_over_70 -2.364e-01 3.348e-01 -0.706 0.480
          rsi_over_80 5.135e-03 4.820e-01 0.011 0.991
          rsi_over_90 7.162e-03 8.650e-01 0.008 0.993
          fourteen_day_rsi 4.193e-04 3.434e-03 0.122 0.903

          Residual standard error: 163.7 on 1581663 degrees of freedom
              (137 observations deleted due to missingness)
          Multiple R-squared: 5.397e-07, Adjusted R-squared: -4.518e-06
          F-statistic: 0.1067 on 8 and 1581663 DF, p-value: 0.999

        • Re:Bad analogy (Score:5, Insightful)

          by professionalfurryele (877225) on Sunday May 25, 2014 @10:14AM (#47087297)

          Sorry but I use both R and python in my work as a biomechanist and while I love working with python and hate working in R, R is not only less verbose for this task, but it is more consistent, intuitive and better documented. Very few languages beat python for simple, easy to read code, but it is not up to the task of doing general purpose statistics. To see why this is the case consider a problem with that blog post. All the diagnostic plots I need to do to check the regression are missing, no qq, no cook's, not even something simple like fitted vs. residual. Now consider what happens when I notice that while the fit is decent the residuals depend on what subject I'm looking at and I need to vary the error term. Or need to switch to a mixed effects model because there is clearly a dependence on the intercept by subject.
          Seriously when i say I hate R, I mean it. The code is ugly, it can be hard to read and woe betide the poor git who makes the mistake of needing a plot more complicated that something lattice can do. It is still better than python for statistics.

        • With judicious use of semicolons, you could fit all that into a single line.

          You might have to scroll horizontally a lot, but it's still a single line!

    • An Argentinian chef is more likely to make great sushi than a Japanese automotive engineer.

      There's an even closer-to-food analogy for this: If you want a good Italian pizza, get a Greek to make it. I have no idea why this works, but the best Italian pizzas always tend to be made by someone called Nikos or Costas.

      • (While you may be right, following slashdot conventions the analogy was intended as a car-analogy, not a food-analogy.)

  • R itself is okay, but even as a long-time user I don't think the language or environment itself is all that much to brag about. What makes it great for statistics is just that statisticians use it, which means that a lot of the packages are written by statisticians. That makes a big difference: recent papers often have R implementations, standard problems have well-maintained R packages for them with all the bells and whistles, etc. As Matloff notes, this means they often have everything that statisticians are looking for, while straightforward textbook implementations you often find in other languages often aren't nearly as thorough in how they handle the statistical models, or only handle some special cases (though there are some really good packages in other languages, just not as many).

    But I don't think that has much to do with R itself being uniquely suited to statisticians. It's used for historical reasons: Bell Labs S was influential in the field way back when nothing like Python or Julia existed, and statisticians started using it because it was a lot nicer than Fortran, which is what other areas of science mostly used back then. GNU R is essentially a free-software workalike for Bell's S, and it's kept most of the community on board through a mixture of existing packages, familiarity, and inertia.

    • by jythie (914043) on Sunday May 25, 2014 @08:44AM (#47086947)
      *nods* who uses a language has more impact on its usefulness then anything inherent to the language. LIbraries, support community, easy of hiring people who both know the language and have domain specific skills, much more important then what kind of sugar the language has.
    • by HuguesT (84078) on Sunday May 25, 2014 @09:13AM (#47087061)

      R has some pretty unique graphing packages. Nothing that I know of matches the way you can do 2D and 3D plots in R. Not Python, not Gnuplot, not Julia, not Matlab, not Excel, not Mathematica, nothing.

      • by Trepidity (597)

        Around here Python's matplotlib has been making some inroads in the plotting category, even among people who use R for the actual data analysis, but it's admittedly not as featureful as the whole suite of R plotting packages.

    • by jonnyj (1011131) on Sunday May 25, 2014 @03:46PM (#47088985)

      Completely right.

      We use R extensively in work. Programmers talk about R's libraries, but that's not the real reason we use it. The killer blow is that the _documentation_ is written by statisticians. That means that it's reliable, easy to understand, and honestly tells you the pitfalls of the techniques you're using.

      We're financial guys who are doing stuff in consumer finance that has rarely, if ever, been done in our field. The statistics aren't particularly advanced, but it's impossible to hire someone who understands the industry and knows the statistics already. Statistics text books tend to either be so basic that you already know what they say, or so advanced that you need a PhD to understand them. On the other hand, much of the R documentation is beautifully simple to read, and comes with brilliant worked examples - albeit from fields that are very different from our own. Whenever we're researching potential new statistical approaches, we find blogs stuffed full of examples written in R.

      In short, the R ecosystem makes you a better statistician. Julia and Python can't offer that.

      • A true "statistical" programming language allows the user to define statistical processes in the language and then compute its statistical properties.
        For example:

        x = random() /* a random number between 0 and 1, uniformly distributed */
        y = x*x
        print(E(y)) /* print the expected value of y */

        R is nowhere near that.

  • Meh (Score:5, Informative)

    by hyfe (641811) on Sunday May 25, 2014 @07:49AM (#47086827)
    Statistics major who programmed Python professionally for a few years (and have a MsC in Comp.Sci) ...

    ... this is all posturing and drama, but good on Prof. Norm Matloff for getting some attention. R is rather usefull, has quite a few extremely usefull features as a language, including some of the best list/indices handling I've seen anywhere. Excellent libraries for statistical work, but it also has quite a few the most downright abhorrent language decision I've seen anywhere ever, with the amazingly poor string handling (for a scripted language) topping that list ( http://www.burns-stat.com/page... [burns-stat.com] )

    Python, C, Mathematica and R all have different strengths for mathematical work / numerical calculations though, and using the best tool for the job is what it's about. As always, what the best tool actually is, is also rather subjective, as which tool will best solve a specific task is always dependent on your skill with the different tools. I do agree with professor though, even though there's quite abit of Python hype (python + scipy/matplotlib is amazing) R is not being replaced anytime soon. It's too good at what it's good at.

    • by Pseudonym (62607)

      [...] using the best tool for the job is what it's about.

      Ah, but from the point of view of a computer scientist, the "best tool for the job" isn't necessarily the best tool that currently exists. R is a fabulous set of well-documented algorithms and linked together with one of the bizarre, poorly-specified and inadequately-documented language with a flaky, abstraction-leaking, poorly-performing implementation [purdue.edu].

      I think it's great that R is written by statisticians for statisticians, and that statisticians fin

    • by PerlPunk (548551)

      I agree with the above about R. But as regards to reliability, I would prefer SAS to R, even though I hate SAS even more than R. Yes, R has lots and lots of features, good documentation, better libraries than any other out there. But sometimes I find discrepancies between R and SAS in performing the same operations, and when I test which is right SAS always seems to win. That is to say that R as an open source platform has the same problems open source platforms tend to have -- buggy code, sometimes inconsi

  • by kav2k (1545689) on Sunday May 25, 2014 @08:03AM (#47086861)

    A joke I've read recently [twitter.com]:

    I'm not sure if "R is written by statisticians, for statisticians" is a good thing e.g. "stadiums are built by footballers, for footballers"

  • by nurb432 (527695) on Sunday May 25, 2014 @08:25AM (#47086907) Homepage Journal

    Use the right tool for the job and stop bashing other tools that were designed for different jobs .

  • by TheRealHocusLocus (2319802) on Sunday May 25, 2014 @09:27AM (#47087105)

    "Arrrr.... fix yar name 'R' while you may, maties!!"

    I may not have the belly for Deep Statistics but I do know abut Internet Search noise levels. I remember trying to do research on WebDAV (believe me, there is such a thing) only to discover that folks discussing it invariably refer to it as 'dav'. Because saying "Distributed Authoring [and] Versioning" out loud makes you spit out your toothpick. Any attempt to search 'webdav' yielded only the sterile official pages, and attempts to search on 'dav' with other keywords brought up conversations from the community of Disabled American Veterans who also use the term in casual conversation, and have said an awful lot over the years. They occupied 'dav' first.

    Now you may think you can pull off a 'C' where Google seems to pick off relevant results if you combine it with any computery term, but it was not always so. It has taken an incredible saturation of C, and perhaps some special coded cases on Google's part, for this to come about.

    The success of Perl is due in some part to the ability of confused people to obtain help and advice about it merely by searching on its unique spelling.

    So the best way to push this R language is with a refit of the name. Go with the pirate theme, it will sell many more T-shirts than those of silly camels and pearls. But stake out a bit of Keyword Real Estate that presently has a relatively low population density.

    Google search result estimate counts, descending order,
    r --- 2,730,000,000
    ar --- 656,000,000
    arr --- 24,400,000
    arrrrrrrr --- 3,060,000
    arrrr --- 876,000
    aarr --- 638,000
    arrr --- 536,000
    arrrrr --- 405,000
    aaarrrrr --- 267,000
    arrrrrr --- 205,000
    arrrrrrr --- 129,000
    aarrr --- 107,000
    aarrrr --- 107,000
    aaarrr --- 56,600
    aaarrr --- 56,600
    arrrrrrrrr --- 52,400

    Adding arrrs is not enough since talking like a pirate is typically accomplished with a single 'a', so ar+ space is pretty well populated up to ar{5}, it looks like best ratio is around a{3}r{3}. But even choosing the less-optimum and easier to type a{2}r{3} by using 'aarrr' instead of 'r' you have improved the signal to noise ratio by a factor of twenty-five thousand.

    Push the name change firmly and decisively. This means that if anyone mentions 'R' there should be immediate responses that ask, "What AARRR you talking about?" This will inject the proper searchable term into the discussion while it reminds the poster of the name change.

    For an interesting 9 minute lecture that might help sell you on this idea, listen here [upenn.edu].

    • For an interesting 9 minute lecture that might help sell you on this idea, listen here [upenn.edu].

      Certificate warnings freak you out? Try this link instead [upenn.edu], now with matching wildcard, calmer seas and less mogul.

    • It is scary sometimes how much control the limitations of Google Search has over our lives.

      For example, the best anti pirating system you can use for any game or film is to name it with less than 3 characters. It then becomes very hard to search for it.

      It took me days to find "9" (and I know others who had similar problems), and I think I never did end up seeing "B".

      • If google search is limiting your pirating, you may want to investigate something a little more specialized. I assume you're talking about the 2009 film with Jennifer Connelly, not the 2005 short nor the video game - either would be two clicks away after less than a minute.

        And if Google Search is really impacting your life in any meaningful way, you should step away from the keyboard for a weekend.

        I think this is more a case where you detected a pattern from two events, and extrapolated to assume that ever

        • Well I specifically mean the specialised searchers.

          Go to The Pirate Bay. Search "9", Search "9 2009".
          Neither of those return any useful results.

          And I guarantee you that that would of effected the number of people who torrented it.

    • I'm afraid your research neglects a huge subset of the Talk-Like-A-Pirate word space, 'yarr' has 523,000 results

  • by Johnny Loves Linux (1147635) on Sunday May 25, 2014 @09:43AM (#47087167)
    Be sure to use RStudio as the front end: http://www.rstudio.com/ [rstudio.com]. Using on R in a terminal is ok, but having the beautiful GUI frontend RStudio makes working with R sooooooo much better! The help system, plots, R markdown (knitr), and inspecting variables in RStudio is so much easier. As far as comparisons go,
    1. R is no competitor to python for writing generic scripts.
    2. Python (numpy, scipy, statsmodels, pandas, sklearn, matplotlib, ipython and ipython notebooks) is not yet ready to compete with R for doing statistical analysis but give Python a couple of more years and then slashdot should do a review of how it compares.
    3. You can always call R from python using the r2py module. This is really easy within an ipython notebook using the %load_ext rmagic command.

    For a nice video on using ipython notebook in data analysis: https://www.youtube.com/watch?... [youtube.com]

    For a nice selection of ipython notebooks for doing various type of data analysis: https://github.com/ipython/ipy... [github.com]

  • by wisnoskij (1206448) on Sunday May 25, 2014 @10:08AM (#47087271) Homepage

    Having seen the state of programming in the Sciences, I really do not thing that "built by statisticians" is something you would want to advertise.

  • by umafuckit (2980809) on Sunday May 25, 2014 @10:09AM (#47087277)
    A few examples are provided in TFA but it's all rather vague as to why R "beats" Python. I've been using R for years for fitting mixed effects linear models. It does this really well, it makes it easy to compare models, it's got all the cutting-edge stuff in it. The problem with R, however, is that it's shitty and unintuitive as a programming language. I do all my pre-processing in MATLAB and I only ever export to R when I have a final data frame that needs a moderately complicated statistical analysis.
  • This guy must have been reading the recent stuff on Fortran and decided to jump on the bandwagon.

    Fortran was written by engineers and scientists for engineers and scientists.
    R is written by statisticians for statisticians.

    Well, there you have it. If a language or other kind of tool was developed by practitioners of X for other practitioners of X, it’s likely that it will be better than some other tool that was designed for a different purpose.

    Who would have thunk it.

  • DSLs (Score:4, Insightful)

    by jbolden (176878) on Sunday May 25, 2014 @12:35PM (#47088051) Homepage

    He's probably right. All other things being equal a good Domain Specific Language will crush a General Purpose Language in its domain. If Julia is much faster than R and that were unfixable it would still be far easier to write a library in Julia accessible by R than to train R users in all of Julia's concepts.

    General purpose languages can sometimes get close to DSLs in effectiveness and then the greater diversity of users creates an economy of sacle and deep entrenchment which drives DSLs away. But then with a large and highly diverse user base the General Purpose language isn't able to rapidly adapt so DSLs spring up to fill niches. Some of those DSLs become incredibly successful and start to move into other domains diversifying their purpose and user base to become General Purpose Languages and the cycle repeats.

  • I'm shocked to learn that a purpose-built programming language might be better at its specific purpose than a general purpose programming language. Shocked I say.

    I'd be even more shocked if a bunch of mathematicians had the good sense to pick a Google searchable name for their language. One PIA thing with C is how hard it is to search Google for documentation when you don't remember the exact function name.

    • by cellocgw (617879)

      I'd be even more shocked if a bunch of mathematicians had the good sense to pick a Google searchable name for their language

      You young punks have any idea by how many years R precedes the existence of google (or even alta vista)? Same goes for the c language, FWIW.

  • We all know Raul Julia as M Bison beats them both. And Raul Julia's reading of "Mystery on the Docks" on Double "R" (Reading Rainbow) lives on in my mind as one of the great renditions.

  • by StripedCow (776465) on Monday May 26, 2014 @05:07AM (#47091329)

    From the summary:

    And R is Statistically Correct

    But Python is correct all the time.

"The whole problem with the world is that fools and fanatics are always so certain of themselves, but wiser people so full of doubts." -- Bertrand Russell

Working...