Forgot your password?
typodupeerror
Programming Software IT Technology

Ruby Implementation Shootout 112

Posted by ScuttleMonkey
from the always-with-the-shooting dept.
An anonymous reader writes "Ruby has an ever growing number of alternative implementations, and many of these attempt to improve the suboptimal performance of the current mainstream interpreter. Antonio Cangiano has an interesting article in which he benchmarks a few of the most popular Ruby implementations, including Yarv (the heart of Ruby 2.0), JRuby, Ruby.NET, Rubinius and Cardinal (Ruby on Parrot). Numerical evidence is provided rather than shear opinions. The tests show that Yarv is the fastest implementation and that it offers a promising future when it comes to the speed of the next Ruby version."
This discussion has been archived. No new comments can be posted.

Ruby Implementation Shootout

Comments Filter:
  • by tcdk (173945) on Monday February 19, 2007 @02:14PM (#18069542) Homepage Journal
    This being /. I don't expect anybody to actually read the acticel, but before you begin your rant^H^H^H^H ... post about how these tests are useless etc. etc. please, at least read the article disclaimer:

    Don't read too much into this and don't draw any final conclusions. Each of these exciting projects have their own reason for being, as well as different pros and cons, which are not considered in this post. They each have a different level of stability and completeness. Furthermore, some of them haven't been optimized for speed yet. Take this post for what it is: an interesting experiment;
    Now fell free to begin your rant about it being a slow news day...
    • by fatphil (181876)
      Unfortunately the guy performing this experiment doesn't know about how to benchmark, in particular how to aggregate results into a mean result.

      In the first table, he uses an arithmetic mean, which is invalid as it unfairly weights slow benchmarks. This has the effect of making the windows version look bad, as against the equivalent linux implementation, the longest benchmark is nearly 30s slower on the windows box, so I was almost tempted to not mention it, and let windows look bad. The correct mean to use
    • by fossa (212602)

      Also note that "average and Median values take in consideration only working tests (they exclude âToo longâ(TM) programs as well)", so the averages for Cardinal and Rubinious with many errors are of dubious utility.

  • Cardinal interesting (Score:4, Interesting)

    by mattr (78516) <mattr.telebody@com> on Monday February 19, 2007 @02:19PM (#18069652) Homepage Journal
    Though not familiar with the project, I'm impressed that the project with by far the most errors (Cardinal), indicating I assume that the least work has been done on it, is still so close to YARV - only 2-4 times slower in a couple tests.

    I can't tell if those fast tests are so trivial that they offer little chance of further speedup, or whether YARV, which has had speed as a goal, is not going to be so much faster than a Parrot-based implementation once it (Cardinal) gets into working on optimization.

    Anyone interested in providing some information on where the YARV performance comes from and whether Cardinal is likely to approach it more closely and farther across the board in the tests?
    • Re: (Score:2, Insightful)

      by archen (447353)
      Personally what annoys me more is that this is giving me no benchmark against how ruby in general is performing. Maybe something is twice as fast, but twice as fast as what? Slow as hell? My understanding is that ruby has always lagged a bit in the performance sector, although maybe that has improved over the last 2 years. I'd be more interested to see additional benchmarks against equivalent programs in Perl5 which is sort of the interpreter execution standard, or maybe something in C.
      • by Azarael (896715)
        That probably wouldn't be too difficult to do using the computer language shootout http://shootout.alioth.debian.org/ [debian.org]
        You have to figure out the equivalent tasks(Mandelbrot seems to be be one) and then flip between the two benchmark pages.
    • by Phrogz (43803)

      "Though not familiar with the project, I'm impressed that the project with by far the most errors (Cardinal), indicating I assume that the least work has been done on it, is still so close to YARV - only 2-4 times slower in a couple tests."

      1. It's easy to run fast when you only account for a very limited subset of functionality. I'd be shocked (am shocked) if preliminary results of something like this *weren't* much faster than the original. I'd take it as a sign that it's not worth it to continue (at lea
    • Re: (Score:2, Informative)

      by chromatic (9471)

      Note that Parrot has had little optimization too.

    • Though not familiar with the project, I'm impressed that the project with by far the most errors (Cardinal), indicating I assume that the least work has been done on it, is still so close to YARV - only 2-4 times slower in a couple tests.

      If your program doesn't have to return the right answer, it's very easy to make it go really fast.

  • by jrwr00 (1035020)
    they need to post the specs of the computer they where using tho, i could post numbers, but i would be running it on a p1 150mhz!
    • by h4ter (717700)
      My tests were conducted on an AMD Athlon(TM) 64 3500+ processor, with 1 GB of RAM.

      It's right there under the large "Benchmark Environment" heading.
  • A shear opinion is like, a comment about Britney's latest screwup.

    Try "sheer".

    • Re: (Score:1, Funny)

      by Anonymous Coward
      When my shears talk, shrubbery trembles!

      An y'all don' wan' nunna my rake-fu!
  • by Jagasian (129329) on Monday February 19, 2007 @02:34PM (#18069870)
    Performance isn't everything, but then again, when you are 400 times slower than Java [debian.org]... performance starts to matter.
    • Re: (Score:3, Insightful)

      by arevos (659374)

      Performance isn't everything, but then again, when you are 400 times slower than Java... performance starts to matter.
      Sure, if you're interested in generating mandelbrot fractals or the spectral norm of a matrix. However, Ruby typically isn't used for such computationally intensive tasks.
      • Re: (Score:1, Informative)

        by Anonymous Coward
        No, but it IS one of the slowest of the "dynamic" languages. Tcl, Perl and Python are all faster, have better threading and i18n support.
        • Re: (Score:3, Insightful)

          by Rufty (37223)
          Yep, I agree. Ruby's slow. Probably slower than the rest at just about everything.
          Except speed of code development.
          So for my one-off scripts that run for 45s in ruby instead of 0.1s in perl, well
          what's mattered to me is that it took 5 mins to write, rather than the 90mins +
          a brain tumor that perl does*

          * I last used perl at about 4.036, tried to get into "objects" with perl 5, and
          jumped to ruby for the sake of my sanity.
          • by drolli (522659)
            Yes, even if i like perl very much to do small text filter/cgi-bin/data aggredgation programs now and then, the "OOP" Features in it give everybody taking OOP seriously instant headaches....

          • by Hercynium (237328)
            It's too bad you left then.

            Perl 5 is when things finally got interesting. I hated Perl 4, but I don't think I could live without Perl as it is today!

            Granted, Perl pales in comparison to Ruby when it comes to "interesting" (and I mean that in a good way! (for ruby))

            Use the right tool for the job. And that depends on the job, the toolbox, and the programmer. For example, for interfacing with our hideous OSS gateway most programmers would use Java. I would use a length of steel rebar and a lot of foul language
    • Re: (Score:3, Insightful)

      by TopSpin (753) *
      Performance isn't everything, but then again, when you are 400 times slower than Java..

      400 times slower? Still too efficient. Consider a Ruby implementation on the JVM and multiply inefficiencies! Should be thousands of times slower for those same benchmarks.

      About JRuby; Sun recently hired [slashdot.org] two (both?) of the JRuby developers and progress has accelerated [codehaus.org]. The promise is a highly capable Ruby implementation running on a JVM. This, coupled with very recent changes to the JVM [artima.com] to facilitate scripting langua
      • by Decaff (42676)
        Consider a Ruby implementation on the JVM and multiply inefficiencies! Should be thousands of times slower for those same benchmarks.

        There is no justification for this statement. One of the aims of JRuby is to compile to JVM byte code, which can then be translated to highly optimised machine code.

        Apparently Sun isn't content to let Microsoft's CLR become the de facto standard bytecode runtime platform.

        Do you realise how many instances of the JVM there are right now? Not just on servers, but on mobile devi
    • by cout (4249)
      We have been using Ruby in a production environment for the last seven years at a company that trades a significant portion of Nasdaq and NYSE-listed securities. We have been using C++ for the performance-critical portions of the system and Ruby for the portions of the system that we want to get out the door more quickly. I won't lie: we've run into more than once case where the Ruby interpreter just wasn't fast enough. However, had we done the initial pass in C++, it could have taken days or weeks longe
  • Any YARV experts (Score:3, Insightful)

    by Timesprout (579035) on Monday February 19, 2007 @02:40PM (#18069948)
    Can anyone comment on why the YARV implementation is so much faster than the standard implementation? I know for example the SUN VM was originally only intended as a reference VM and faster implementations were developed but not on the scale of the differences highlighted here.
    • Re:Any YARV experts (Score:4, Informative)

      by Balinares (316703) on Monday February 19, 2007 @03:01PM (#18070298)
      Short answer: various things, compilation to bytecode before execution (I thought it was the case in the current interpreter, but might have been mistaken), etc.

      Slightly less short answer: if I'm not mistaken, YARV includes a JIT compiler, similar to what Psyco [sourceforge.net] does in the Python world. Psyco has been known to accelerate code execution up to 100x times, so I'd expect YARV to be even faster than this benchmark shows when it's stabilized.
    • by mo (2873) * on Monday February 19, 2007 @03:17PM (#18070576)
      The standard 1.8.X Ruby interpreter is a single-pass interpreter, and YARV is a virtual machine implementation. You can expect big improvements when moving to a VM implementation from an interpeter. The reason Java has never had such big improvements is that it's always been based on a VM.

      This rule isn't exactly hard and fast, as verying implementations of VMs and interpreters can have different performance characteristics. For example, while perl is still probably considered an interpreted language, it's quite fast due to the interpreter using many compiler tricks such as parse tree optimizations. The ruby interpreter however has been notoriously slow, which is why ruby people are so excited about it.

      • Java had this jump twice, once when the vm went from purely interpretet to jit compilers, the second one was when the vm went from jit compilers to runtime optimization + jit.
  • unicode? (Score:3, Interesting)

    by bcrowell (177657) on Monday February 19, 2007 @03:15PM (#18070536) Homepage
    What's up these days with ruby and unicode? Of these implementations, do some do it one way and some another?
    • Ruby and Unicode (Score:3, Interesting)

      by metamatic (202216)
      What's up is that there's massive disagreement on whether Ruby's standard strings and characters should become Unicode. There are quite a few people used to the old world where a string was interchangeable with a vector of 8 bit bytes, and they don't want to let go [archivesat.com]. To add to the problems, Unicode is controversial in Japan [jbrowse.com]. So, expect Unicode to continue to be painful and inconsistent in Ruby even after the next major release.
      • by tuffy (10202)
        Why don't they implement a Unicode string which can be encoded to a classic string via UTF-8 or some other desired encoding. Then add an decode() method to regular 8-bits-per-character strings to reverse the process. This eliminates ambiguity between "a binary blob of random data" (regular strings) and "human-readable text in some language" (Unicode strings). Python's used this method for years without a problem, so Ruby should be able to follow suit.
      • > There are quite a few people used to the old world where a string was interchangeable with a
        > vector of 8 bit bytes

        No. No, no, no. NO. Please, no. Seriously. This is bad. A LOT of damage happens because people can't be bothered to spend the 10 minutes it takes to learn the basics of Unicode. It's easy. I don't always think very highly of what Joel Spolsky writes, but his introduction to Unicode [joelonsoftware.com] is an ABSOLUTE must-read.

        Seriously. Strings are NOT interchangeable with byte arrays. EVER. Without an enc
        • by metamatic (202216)
          Yes, well, as you'll see from the ruby-core thread I linked to, I'm on your side. I don't get why anyone would even want to use String objects for binary processing rather than something actually suited to the task, but I thought I'd at least offer a possible compromise... let the language move forward with Unicode Strings, but have a SimpleString that was Unicode-unaware that you could pretend was a binary vector, for people who insist on it.
    • There's but one language you'll find with rigorous support of unicode to make it a non-thought, and that's tcl. Add to unicode the excellent wiki [wiki.tcl.tk], a grammar completely defined in eleven rules [www.tcl.tk], and a versatile cross-platform GUI toolkit [wiki.tcl.tk], and you've got a language that can't be beat for a great many situations.

      Suggest you to start with this page [wiki.tcl.tk] to get a glimpse of what Tcl is all about!
      • Say what you will about Micrsoft, but they do it better than just about anyone else as far as internationalization goes on the .NET platform. They go beyond pervasive support for Unicode: the framework has built in support for all standard ISO cultures, and can automatically translate and format dates, numbers, currencies, etc. on the fly with almost zero extra code.

        Of course you have to manually translate text strings into resource files for each language, but once you do that, programmatically switching b

  • I don't think anyone decides to use Ruby based on performance, really. I'd be a lot more interested to hear whether any new features (notably, proper Unicode strings and native threads) are included in any of these implementations -- I really can't use Ruby without at least the former. I would have thought the Java and .NET implementations would be able to do these without too much trouble -- I'm well aware of the difficulty of working around the issues in the original C implementation but they are all ju
  • I would use Ruby more if the runtime performance were better. For speed and dynamic language niceness I use Franz Common Lisp for most of the work for my current customer (AI medical applicaion) and I still find myself using Java a lot for serverside deployment goodness :-)

    I would have prefered Ruby, BTW, because it is easy to train people in Ruby development while Common Lisp is a harder learning curve.

    Once Yarv is very stable (maybe it is already?), then I will use Ruby for a larger percent of my new proj
    • I just built the Ruby 2.0 svn trunk under OS X (as per directions on the web, I used an application suffice of -yarv to not interfere wih my production Ruby build).

      My irb-yarv will not work (for me), but I have tried running ruby-yarv against the examples for my in-progress "Ruby AI" book and did not see any problems.

      Yarv will be cool! Especially when it supports Rails.
  • This study backs up what everyone who has tried Ruby on Windows already knows... it stinks! I'm actually not surprised to see that some of their tests errored out completely on Windows, even with the Ruby 1.8.5 binary.

    Granted, Ruby is mostly used with Rails right now, but you'd think that if the language proponents want the language to take off, Windows support would be taken more seriously. In the couple years since I first tried Ruby, it doesn't seem like it has improved much at all.
    • by Stamen (745223)
      I think this is just because of the age of the language, as the same was true for Perl and others when they were young.

      Most servers are going to be running a *nix, and the primary domain of scripting languages is the command line and the server. If someone is running Windows as a web-server, they most likely will be using .net, because frankly, why else would you be using Windows as a web-server? (Serious question)

      As Ruby matures, so will things like performance and secondary OS support. JRuby and Ruby.n
      • Re: (Score:1, Insightful)

        by Anonymous Coward

        I think this is just because of the age of the language, as the same was true for Perl and others when they were young.
        Ruby has been around for 12 years. It's only 7 years younger than Perl, and just 4 years younger than Python.

        Perl, for one, acquired decent Windows support in the late 1990s... at a time when it was younger than Ruby is today.
      • Re: (Score:3, Insightful)

        by shmlco (594907)
        "As Ruby matures, so will things like performance and secondary OS support."

        And when will that happen, exactly? I've been hearing about R/RoR for years now, and it's still distinctly in the low-performance category. Yet every time I go visit the site for updates all I see is talk about how they're adding feature X and Y in the next point release.

        These guys need to stop dinking with the language, freeze it, and work on fixing bugs and increasing performance so people out here in the real world can actually u
        • Re: (Score:2, Interesting)

          by zallus (714582)

          make it a stable and solid development platform...
          Whoever said that was their primary, or even secondary goal? I use ruby as a prototyping language before rewriting in C. I only use it because it's so easy to work with all the "features of the day" in it.
      • Re: (Score:2, Insightful)

        by chromatic (9471)

        I think this is just because of the age of the language, as the same was true for Perl and others when they were young.

        Ruby's almost 14 years old! By 2001, Perl had fantastic Windows support.

        Now people may deploy Ruby or Perl or Python or PHP applications to Unix and Unix-like servers, but I know plenty of people who develop on Windows. Some of them even have a choice.

        • Re: (Score:2, Informative)

          by Stamen (745223)
          12 years old (December 21st, 1995) but who's counting a few years.

          First off, the parent of my post said that the Windows port wasn't as fast as Linux and should be supported and worked on more. Ruby supports Windows just fine thank you.

          Second, Ruby has had an interesting history. It was basically unknown in the west until Dave Thomas and Andy Hunt wrote the first book in English in 2000. At that time, the language started to take off, and only in the last few years has development really ramped up. So t
          • by chromatic (9471)

            12 years old (December 21st, 1995) but who's counting a few years.

            Matz, I suspect, who started in February 1993, if I recall correctly.

            It was basically unknown in the west until Dave Thomas and Andy Hunt wrote the first book in English in 2000. At that time, the language started to take off, and only in the last few years has development really ramped up. So technically it is 12 years old, but in reality it is a very new language.

            Immature, maybe, but it's about the same age as Java.

    • Ruby and Ruby on Rails seem to appeal largely to Mac users. The Mac-only textmate text editor is extremely well integrated with Ruby and Rails, and I believe almost all the principal developers of Rails work on Macs. Apple includes Ruby in their standard OS distribution.

      I think the "it just works" philosophy of the Mac has a lot in common with the similar philosophy of Ruby and Rails. So if you are attracted to Rails you are likely to be attracted to Apple products and vice versa.

      Since Ruby adoption is l
      • by shmlco (594907)
        "...and amazing.com was developed (and is being developed) using Ruby on Rails"

        Gack! Somebody needs to hire a graphic designer quick. That site is NOT a promising endorsement for RoR...
        • Alas, as you probably know, most developers are not artists. I'm a developer at heart who's trying to keep control over the site, and avoid spending money I don't have by designing it myself.

          The big problem I have, though, is that most sites I see created by designers seem to follow the same rather boring mold. Every site developed in the last year or two looks just like every other site developed in the last year or two.

          I'd appreciate knowing what it is that creates such a negative reaction to what I've
          • by shmlco (594907)
            Why do half the links on the bottom go to pages with no format? Why do About us and Safety Tips use different layouts, widths, line widths, etc. Why the 1980's-style colored text on a patterened background on the home page? Why is the page style/font/color scheme different from the logo scheme, which is different from the sign-up block, which is different from the sign-up button.

            You have three choices: 1) hire a designer; 2) get a template; 3) find a designer.

            You've already indicated you don't want to do nu
    • ...why do you care about Windows support?

      And if your server is Windows, I'd say you're on your own. If, for some perverse reason, you want it to work well, the code is there, but why should we help you?
      • by trimbo (127919)
        ...why do you care about Windows support?

        I do not want to deploy Linux on my home machines, but a couple of my sites are hosted on Linux with Rails. It would be nice to be able to work locally with the same apache-mysql-ruby config, even though I don't want to install Linux at home. Ruby also would have probably made more inroads into my Windows-centric workplace if it had better support, but instead I went with ASP.NET after failing to get Rails on Windows to work effectively on Windows with Apache a
        • by Stamen (745223)
          60,000 downloads of Ruby for Windows last month on RubyForge says someone is getting it to work on Windows. What does that mean "work effectively"?

          I highly doubt a Window-centric Corporation would retrained all thier developers, and switch to an open source laungauge that isn't supported by Microsoft only if it had "better support". I call shanigans. Ruby is making inroads in PHP, Perl, and Python shops; I woudln't hold my breath for Windows shops. I have many clients like this, they rarely even know ab
        • I do not want to deploy Linux on my home machines, but a couple of my sites are hosted on Linux with Rails. It would be nice to be able to work locally with the same apache-mysql-ruby config, even though I don't want to install Linux at home.

          You could mess with it on the server, which was my point. What makes your local Windows machine more convenient?

          I could make this into a big deal about you refusing to roll out Windows at home, but I imagine this is closer to a minor inconvenience than a major stumbli

        • I'm curious why you felt compelled to set up Rails with Apache on your home systems. I've always found the pre-installed WebBrick server better for home development.
          • by trimbo (127919)
            A good question! I simply wanted to have an environment that was more akin to what the server was running to get a better idea of performance and such. When doing Python development for the web, I've always had the same setup on my Windows box and server (Apache + mod_python + clearsilver).

            Anyway, to the other posters: Don't shoot the messenger telling you that Ruby sucks on Windows. As I said, TFA points it out very well by having tests that crash out completely on Ruby 1.8.5 for Windows. It's not g
  • by zymano (581466) on Monday February 19, 2007 @04:41PM (#18071732)
    C and cgi.
    • C and cgi.

      Amen. With critical functions (XML parser, for instance) rewritten in assembly for speed.
    • Re: (Score:2, Insightful)

      by FrnkMit (302934)
      I worked briefly at a shop that used C and C++ for CGI. They were still struggling with memory errors in their in-house string libraries.

      But really, Ruby's for more than just Web stuff. (As is Python and the rest.)
      • by quigonn (80360)
        I worked briefly at a shop that used C and C++ for CGI. They were still struggling with memory errors in their in-house string libraries.

        And they deserve it. Everyone who uses C++ but not the std::string class it comes with does so.

      • But really, Ruby's for more than just Web stuff.


        Amen. I'm not the biggest fan of RoR - I don't find it all that fast unless you're building a ground-up app, and even then it bugs me sometimes.

        Ruby for other things - text parsing and the like is just awesome. I have a hard time with some doofuses in my office explaining them that Ruby on Rails != Ruby...
    • Often overlooked: C is a system programming language and not an application programming language. C was designed to write the Unix operating system and quite honestly it is not good for anything else.

      Yes you can use C for all sorts of other programming needs (if you have a masochistic tendence) - all programming needs in fact as C is turing complete - but it isn't any good at it.

      Martin

      Note: I have learned C in 1990, updated my C knowledge regularly, own a copy of the C99 ISO standard and I have quite a lot

Gee, Toto, I don't think we're in Kansas anymore.

Working...