Forgot your password?
typodupeerror
Programming Software IT Technology

Twitter On Scala 324

Posted by kdawson
from the going-off-the-rails dept.
machaut writes "Twitter, one of the highest profile Ruby on Rails-backed websites on the Internet, has in the past year started replacing some of their Ruby infrastructure with an emerging language called Scala, developed by Martin Odersky at Switzerland's École Polytechnique Fédérale de Lausanne. Although they still prefer Ruby on Rails for user-facing web applications, Twitter's developers have started replacing Ruby daemon servers with Scala alternatives, and plan eventually to serve API requests, which comprise the majority of their traffic, with Scala instead of Ruby. This week several articles have appeared that discuss this shift at Twitter. A technical interview with three Twitter developers was published on Artima. One of those developers, Alex Payne, Twitter's API lead, gave a talk on this subject at the Web 2.0 Expo this week, which was covered by Technology Review and The Register."
This discussion has been archived. No new comments can be posted.

Twitter On Scala

Comments Filter:
  • by 0100010001010011 (652467) on Tuesday April 07, 2009 @04:38PM (#27495261)

    Kidding aside, is this a 'nail' in the coffin of scalable Ruby? 5 years ago people were saying the same thing about PHP scaling but Facebook has done a rather nice job of making it scale. Twitter was supposed to be the poster child of how awesome Ruby and RoR was.

    Difference is, Facebook is still using php, Twitter is going toScala.

    • It's funny you mention that on a day that facebook has been up and down like a yo-yo.

    • Re: (Score:3, Insightful)

      by Foofoobar (318279)
      Never was a nail except for the Ruby community that was in denial. Ever developer in the world except Ruby fanboys knew Ruby's inherent problem was scalability. Benchmarks showed it but they would always slant their own benchmarks to show the opposite. But facts are facts and in the end you can't deny the truth. So this is where we are at. The question is will Ruby fanboys still choose to deny the issues with Ruby or accept that it does have inherent limitations?
      • by K. S. Kyosuke (729550) on Tuesday April 07, 2009 @04:52PM (#27495517)
        Ruby is a language. Languages usually don't have problems with scalability.
        • Re: (Score:3, Informative)

          by digitig (1056110)
          Really? Try comparing the scalability of, say, FORTRAN or COBOL against any modern language with decent encapsulation.
          • "Decent encapsulation?" Don't know what that is supposed to mean here, but Common Lisp, e.g., fares quite favourably against Fortran.
        • by tcopeland (32225) <tom.thomasleecopeland@com> on Tuesday April 07, 2009 @05:08PM (#27495761) Homepage

          > Ruby is a language. Languages usually don't have problems with scalability.

          Quite right. An application with 8 million users will have scalability challenges regardless of what type of language opcodes are being executed. At some point it's all about architecture.

        • by Foofoobar (318279)

          Ruby is a language. Languages usually don't have problems with scalability.

          So you are saying Visual Basic and C are going to have the same scaling challenges and that they scale equally well? Well all I can say it pass that magical crack pipe cause I want a toke too.

          • Re: (Score:3, Insightful)

            by K. S. Kyosuke (729550)

            Visual Basic was so stupid that Microsoft could have easily made it almost as fast as C - if only they had wanted. I guess they had other priorities.

            I must ask one thing: Have you ever heard of Self 93? They made it so good fifteen years ago that it topped at half the speed of C...a language where everything (!) was a message send and *anything* could have been redefined at *any* moment.

            If someone threw the necessary money at Ruby, it could have easily the performance of a modern Smalltalk implementation,

            • Re: (Score:3, Interesting)

              by david.given (6740)

              If someone threw the necessary money at Ruby, it could have easily the performance of a modern Smalltalk implementation, as Ruby is essentially Smalltalk in diguise.

              You might be interested in StrongTalk [strongtalk.org]. It's a Smalltalk-80 rewrite with optional strong typing and pervasive JIT, meaning that it's incredibly fast and robust. It is, unfortunately, Windows only at the moment, but it's all BSD licensed --- VM, image, source browser and all --- and they're looking for people to help with a Linux port.

        • by hhr (909621) on Tuesday April 07, 2009 @08:04PM (#27497693)

          The article is about Ruby on Rails. Ruby on Rails is not just a langauge. It is a lanaguage and a web framework. Frameworks very much affect your scalability.

      • by mini me (132455)

        Everyone knows that Ruby has performance issues. The fact is that the performance problems are not problems for short-lived operations like serving web pages. The article even backs up the claim.

        Ruby scales just as well as every other language. The article never once claimed that Ruby does not scale. It claimed Ruby had performance problems; a topic mostly unrelated to scaling.

        If Ruby users were in denial about the problems with Ruby, there would not be several projects working on solving the problems. Howe

        • by Foofoobar (318279)
          And where Facebook and PHP can accomplish this goal for the frontend and the backend (and many other corporate and enterprise PHP sites), Ruby can merely do frontend work. That doesn't seem like scalability to me. That seems very limited. That to me says 'as long as you don't TAX the engine, it can perform'. That's not scaling. That's getting by.
          • by caramelcarrot (778148) on Tuesday April 07, 2009 @06:34PM (#27496839)
            Facebook doesn't use PHP for the backend, it's mostly C++, Python and Erlang.
        • Re: (Score:3, Interesting)

          by lotzmana (775963)

          I agree, but wish to add a comment about vertical and horizontal scaling.

          Ruby and Python have poor multi-threading. They don't scale well on multi-CPU platforms.

          from the interview:
          Robey Pointer: Green threads don't use the actual operating systemâ(TM)s kernel threads.

          So, a Ruby application can't scale well vertically -- one can't just upgrade the machine with more CPUs for example.

          At the same time, no language is inherently prohibiting horizontal scaling, if application design provides for it

      • There you go again! (Score:5, Informative)

        by Jane Q. Public (1010737) on Tuesday April 07, 2009 @05:18PM (#27495909)
        Ruby does not have a problem scaling. Neither, for that matter, does even Rails. (As the companies that run Basecamp, Campfire, LinkedIn, Lighthouse, and many others will tell you.)

        The fact is that the Twitter folks tried to write their own message queue in Ruby [unlimitednovelty.com], when there was absolutely no reason to do so: there were plenty of pre-made message queues already available for Ruby, and already optimized. Not only did they choose to write their own, unnecessarily, they did it badly [obiefernandez.com].

        And not only that, but Alex Payne has a hidden agenda: he is trying to push Scala to boost interest in the book about Scala he just wrote! [obiefernandez.com]

        Please get some facts before digging up this long-dead and well-buried "Ruby or Rails doesn't scale" bullshit again.
        • by Radhruin (875377) on Tuesday April 07, 2009 @05:40PM (#27496199)
          Anyone who thinks Ruby [hulu.com] on [amazon.com] Rails [zvents.com] can't [scribd.com] scale [yellowpages.com] is as dogmatic in their anti-hype as the original hypers were. The right tool for the right job and all that.
          • Re: (Score:3, Informative)

            Not a single site you posted is anywhere near the volume of traffic that Twitter's seen. Hulu's the closest, but the heavy lifting is done outside of RoR.
        • by mckinnsb (984522)
          Mod parent up. He is dead on.
        • by tieTYT (989034) on Tuesday April 07, 2009 @08:38PM (#27497981)

          Jane Q. Public: Either you didn't read the comments of that blog or you're spreading FUD. Here is a comment from Alex Payne from that article:

          Hoo boy. First of all, I hope you've had a chance to read my general reply to the articles about my Web 2.0 Expo talk [1] and this response to a vocal member of the Ruby community [2]. I sound like a pretty unreasonable guy filtered through the tech press and Reddit comments, but I hope less so in my own words.

          Secondly, the quote at the top of your post is from my coworker, Steve Jenson, who's been participating in the discussion on this post.

          On JRuby: as Steve said, we can't actually boot our main Rails app on JRuby. That's a blocker. Incidentally, if you know of anyone who has a large JRuby deployment, we'd be interested in that first-hand experience. If you don't, it might be a little early to say it would solve all our problems.

          It's also incorrect to say that the way JRuby and Scala make use of the JVM is exactly the same. Much like our other decisions haven't been arbitrary, our decision to use Scala over other JVM-hosted languages was based on investigation.

          On our culture: if you'd like to know about how we write code, or how our code has evolved over time, just ask us. We're all on Twitter, of course, but most of the engineers also have blogs and publish their email addresses. There's no need to speculate. Just ask. There's not a "raging debate" internally because we make our engineering decisions like engineers: we experiment, and base our decisions on the results of those experiments.

          It's definitely true that Starling and Evented Starling are relatively immature queuing systems. I was eager to get them out of our stack. So, as Steve said, we put all the MQ's you think we'd try through their paces not too long ago, and we knocked one after another over in straightforward benchmarks. Some, like RabbitMQ, just up and died. Others chugged on, but slowly. Where we ran into issues, we contacted experts and applied best practices, but in the end, we found that Kestrel fit our particular use cases better and more reliably. This was not the hypothesis we had going into those benchmarks, but it's what the data bore out.

          We get a lot of speculation to the tune of "why haven't those idiots tried x, it's so obvious!" Generally, we have tried x, as well as y and z. Funnily enough, I was actually pushing to get us on RabbitMQ, but our benchmarks showed that it just wouldn't work for us, which is a shame, because it advertises some sexy features.

          Personally, I'm extremely NIH-averse; I research open source and commercial solutions before cutting a new path. In the case of our MQ, one of our engineers actually wrote Kestrel in his free time, so it was bit more like we adopted an existing open source project than rolled our own. Pretty much the last thing we want to be doing is focusing on problems outside our domain. As it so happens, though, moving messages around quickly is our business. I don't think it's crazy-go-nuts that we've spent some time on an MQ.

          I hope my colleagues and I have been able to answer some of your questions. As I said, in the future, please consider emailing us so we can share our experience. Then, we can have a public discussion about facts, not speculation. Perhaps, as commenter sethladd suggested, the onus is on us to produce a whitepaper or presentation about our findings so as to stave off such speculation. Time constraints are the main reason why we haven't done so.

          [1] http://al3x.net/2009/04/04/reasoned-technical-discussion.html [al3x.net]
          [2] http://blog.obiefernandez.com/content/2009/04/my-reasoned-response-about-scala-at-twitter.html#IDComment18212539 [obiefernandez.com]

      • by vlm (69642)

        Never was a nail except for the Ruby community that was in denial.

        Well, yeah, no kidding. Thats because all I see is an endless circle of people repeating what they heard other people saying:

        1) All the cool kids (us) know it won't scale. If you disagree, then you're not cool. Ha ha, thats our proof it doesn't scale.

        2) I don't use it, never used it, never will use it, don't know anyone using it, but I am certain it won't work.

        3) If it won't work on the worlds biggest websites, I can't use it on the departmental intranet, because I'm as cool as the worlds biggest website

    • by julesh (229690) on Tuesday April 07, 2009 @04:50PM (#27495481)

      Difference is, Facebook is still using php, Twitter is going toScala.

      PHP was a mature environment when facebook was launched. RoR was (and still is to a certain extent) a fad environment, popular primarily because of its differentness. People who build sites on a platform because it's the latest thing are less likely to stick with that platform than people who choose a platform that has a solid reputation but is boring. Scala, at a guess, is going to be the next fad platform. Like Ruby, it has some interesting ideas behind it, but it needs a lot of development before we can consider a stable platform for serious applications, I think.

      • Re: (Score:2, Interesting)

        by Radhruin (875377)
        I find your assertion of differentness being the main reason to use Ruby on Rails to be somewhat offensive let alone uninformed, as it suggests that the multitudes of developers using it are doing so not because of technical merits but because they're buying in to some image of differentness. A cursory examination of the typical Rails project and developer should indicate otherwise. Because you don't find it helpful in your work doesn't mean others don't find it helpful or see real benefits from using the s
      • by iluvcapra (782887) on Tuesday April 07, 2009 @05:43PM (#27496247)

        RoR was (and still is to a certain extent) a fad environment, popular primarily because of its differentness.

        Huh, I generally use it because it has really good ORM and migrations, and I really like the syntax (coming from Objective-C it's pretty slick. I also used the PHP language when I was starting out, but one day it tried to insist that $myArr[0] and $myArr["0"] actually pointed to the same object, and I have refused to deal with it ever since; I also got tired of typing str_sub_case_insensitive_for_real_safe(haystack, needle) -- or is it needle, haystack? And is this one of those prank functions that fails to substitute the value but still returns a value that evals true? Or if I leave out one of those underscores, am I in fact calling a function that behaves almost exactly the same way but fails under difficult-to-reproduce circumstances? Maybe they've fixed this and the other sundry atrocities? Maybe they've stopped trying to make it into Perl, as compiled by a C++ compiler, and tried fashioning it into an actual dynamic language? I know, I know, some people like PHP, but I think arguments for the superiority of PHP over Ruby (or Python or Scala or Lisp or WebObjects or Perl6 or really anything else) are going to rest completely on the skills of the Zend interpreter writers, and almost never on the quality/readability/maintainability of the code, or the ease of the development process. You can write good safe code in PHP, that is true, but it isn't very ergonomic.

        You know, RoR is really good at replacing those old Paradox and FMP database systems. I can see how Facebook might prefer PHP, but people trying to replace little inventory/business processes systems generally only need to support a few dozen users, and don't have an army of developers to keep it running. The Universe is big enough to accommodate the utility of Ruby on Rails and the Twitter developer's stupidity.

        • Re: (Score:3, Interesting)

          by gullevek (174152)

          As much I do like coding PHP I completely agree with you. The fact that $arr[0] is the same as $arr['0'] is just insane and that each function has a different order of needle, haystack, replace, etc is just unbearable ... So much that I already have wrapper functions so they are all the same ...

          But unless I find years of time and unlimited money there is no way I can ditch all my php code and go to ruby or python or anything else ...

          • Re: (Score:3, Insightful)

            by iluvcapra (782887)

            The fact that $arr[0] is the same as $arr['0'] is just insane

            I'm not even sure if that was what it was, I think the problem I was having on top of that was that I would pop something off the head of $arr and sometimes it would be $arr[0] and sometimes it would be $arr[1], and this is crazy, but I dug in and low and behold I was actually storing values in the array to ["0"] and ["1"], and since arrays and hashes are the same thing in PHP (this is the real sin), it did what it thought was right, and created

      • by DuckDodgers (541817) <keeper_of_the_wolf@yaho o . c om> on Tuesday April 07, 2009 @08:05PM (#27497701)
        Scala has the significant advantage that it's built on Java and interoperable with Java. Scala source code compiles directly into .class files. You get the speed of the JVM (which is acceptably quick these days), the ability to easily call Java APIs from within Scala, and the ability to run your Scala code on any machine with the JVM.

        It's popular to dislike Java, and even as a well paid Java developer I'm not a huge fan of the language. But Java still is extremely common, and you can even write Java code for your Scala code to use while you're learning Scala.

        Scala also keeps Java's strong static typing and adds functional language features. I don't think it needs any development at all to be adapted for mainstream use.

        On the other hand, as a C++ developer I found learning Java to be child's play. The learning curve from Java to Scala, for me at least, is noticeably steeper. If anything kneecaps Scala I suspect it will be the barrier to entry, not the language itself.
        • Re: (Score:3, Interesting)

          Scala also keeps Java's strong static typing and adds functional language features. I don't think it needs any development at all to be adapted for mainstream use.

          Scala is a great thing, now what it needs is equally great tooling (i.e. IDE support, including refactoring, on the same level as we have for Java). And it's getting there - there is an Eclipse [scala-lang.org] plugin, and a NetBeans [netbeans.org] one - but it definitely needs more work and polish.

          The payoff would be huge, though. Right now, as far as languages go, C# is far more advanced than Java. But Scala is equally more advanced than C# (the only thing on .NET that could compare with Scala is F#, and that's less stable and mature cu

    • by mini me (132455) on Tuesday April 07, 2009 @04:51PM (#27495491)

      While Facebook uses PHP where Twitter uses Rails, Facebook uses a plethora of languages to make the whole system work. So Twitter really isn't going to Scala any more than Facebook is going to Erlang. Which is the say that they use the best tool for the job, not one tool for every job.

    • 'Nuff said. Those twitter posts were once described as "internet SMS messages", didn't they? Short messages...phone systems and heavy networking...Ericsson...Erlang...
    • Re: (Score:3, Interesting)

      by CMonk (20789)
      Huh? Are you confused? Both companies made the stupid mistake of using a web scripting language to do backend heavy lifting. Twitter is fixing that with Scala. (leaving RoR on the frontend because its really good at web frontends) Facebook already fixed that with ERLANG, not PHP. (leaving PHP on the frontend because of the technical debt they've accumulated) http://www.facebook.com/eblog [facebook.com] BTW, PHP is the worst language ever. That is a fact, not my opinion.
  • Stupid (Score:2, Funny)

    by AKAImBatman (238306) *

    They should have just used Java. Wait--

  • by snowraver1 (1052510) on Tuesday April 07, 2009 @04:41PM (#27495323)
    Seriously.
    • Re: (Score:3, Interesting)

      by ajs (35943)

      Well... just to name a few:

      • Anyone watching which companies are growing during the recession
      • Anyone looking at how the latest high-volume services are building infrastructure on the Web
      • Neil Gaiman [twitter.com]
      • Demi Moore [twitter.com]
      • The President of the United States [twitter.com] (though to be fair, his status isn't updated recently
      • Al Gore [twitter.com]
      • John McCain [twitter.com] (currently updating from Hong Kong)
      • John Battelle [battellemedia.com], whose insights about the search industry are often enlightening

      This, of course, does not make Twitter a panacea, but it certainly makes it int

      • Re: (Score:2, Insightful)

        Wow, Demi Moore? Really? Now that's an influential user!
        • by ajs (35943)

          Wow, Demi Moore? Really? Now that's an influential user!

          Not really, but given that I was trying to give a sense of the breadth of the people interested as users, commentators and misc. interested parties, I think she fits in the list just fine.

      • by Dynedain (141758)

        Having a marketing team that makes tweets for you is completely different than being a twitter user yourself.

        Of the people you listed, how many subscribe to twitter feeds?

      • The only one in that whole list I am interested in is Neil Gaiman.
      • Anyone watching which companies are growing during the recession
        ... Should get their info from a more complete source.
        Anyone looking at how the latest high-volume services are building infrastructure on the Web
        .. I'll give you that
        Neil Gaiman [twitter.com]
        Who? (Turns out to be some author) **Update! I just wrote another page!
        Demi Moore [twitter.com]
        Who cares?
        The President of the United States [twitter.com] (though to be fair, his status isn't updated recently
        I would prefer something with mor
  • Scala looks and feels like Java with a tiny bit of Python thrown in for good measure. I'm really not certain why anyone would use it over, say, groovy or just plain Java.

    The really odd part is trying to imagine Scala as a reasonable replacement for Ruby or any other higher level language.

    • by geekoid (135745) <dadinportland AT yahoo DOT com> on Tuesday April 07, 2009 @04:47PM (#27495421) Homepage Journal

      It's new! It's hot! it's what all the kids are doing! It's what crappy programmers can pretend to adapt to instead of writing solid code!

      weeee!!!

      • Re: (Score:2, Insightful)

        by hedleyroos (817147)

        I'm confused - how does a language "scale"? Can you suddenly have 1 billion items in your array instead of 100 million? If that is indeed the case then like the parent said: it is crappy coding and/or design.

        I'm fairly confident you can do twitter in whatever language you want and if designed properly it will scale in the proper sense of the word.

        • A language scales if the language itself (not the "runtime" or whatever) can grow with user constructs to suit the ever-changing needs. For instance, in Python, if you write
          l = [a]

          you will end up with a Python built-in list. Squared-brackets are part of the language syntax and will always refer to Python builtin lists.

          In Scala, which is meant to be a scalable language (as the name implies: SCAlable LAnguage), if you write
          val l = a :: Nil

          you will end-up with a Scala list only because the ::' "keyw
          • Re: (Score:3, Informative)

            by Bruce Perens (3872) *
            This is the same for Ruby. You can create a list, or you can create some object you wrote in hand-optimized C which just responds to the same operations as a list.
        • I'm confused - how does a language "scale"?

          "Scalability" is a multi-faceted term. Most people think of "vertical scaling of the servers" when they hear the term "scalable". Which is to say that the code can handle a higher transaction load on a beefier server.

          But there is quite a bit more to scalability than that. There's horizontal scalability of the servers. i.e. Does the software support plugging more boxes in to handle a greater load? Then there's the development scalability. i.e. Are the concerns of th

      • by A.K.A_Magnet (860822) on Tuesday April 07, 2009 @04:58PM (#27495615) Homepage
        I read between the lines that you call C or C++ solid-code, and if I'm not mistaken, you will find that the kids are doing Scala because the code is more solid. Scala benefits from a typing system close to OCaml's which makes Scala code very, very solid -- especially if you keep away Java specifics (such as nullable objects) in your code and take special care when interacting with Java libs that may do so.

        If I'm mistaken and you're not talking about C/C++, I hope you are not talking about dynamic languages which offer no guarantee whatsoever; you know as a developer I enjoy actually spending my time on working on the business side of my application -- and how to make it scalable, rather than working on low-level specifics and on testing if every pointer is null before dereferencing them. A type system that does this for me (which Scala or ML's parametrized type Option allows) is a bliss.

        Now, I'm not enumerate every language under the sun to see what code you call solid, I guess your answer would be that the code is solid whatever the language it's written in. In the end, it all comes down to binary instructions, right? The question is: how many guarantees do the tools give you? In the case of Scala's compiler, it gives you a lot AND offers you a very enjoyable, lightweight yet powerful syntax.
    • Re: (Score:2, Insightful)

      by Anonymous Coward

      Scala compiles to Java Bytecode and runs on an unmodified Java Virtual Machine. It can call any Java Library and you can write code in Scala that is callable from Java.

      Scala offers much more than simple syntactic sugar. Take a look at their documentation. Especially the nice support for embedded DSLs is VERY interesting.

    • Or just jruby which has proven itself to be quite able to handle under load. From what I understand they had issues with their Message Queue which could have benefited from the native threads in jruby. But we are all speculating on what really is going on with Twitter.

      And what does scala have over say erlang for concurrency and performance?

    • What? Scala is Java mixed with OCaml -- you get an extremly powerful typing system, but it feels like a "dynamic language" such as Python or Ruby. With the performance of Java. Under the hood, it's a brand-new language, very different from all those: it merges functional and object-oriented programming. Yet, for the regular programmer, it feels like Ruby... until he gets used to more powerful features and learns how to designs more complex libraries as embedded DSLs. All that while running on the JVM and
    • by nuttycom (1016165)

      The combination of static typing, type inference and concise syntax for higher-order functions give you a lot that Groovy and Java just fall short on.

      In Java's case, it's the simple fact that you can't reasonably implement HOF's like filter, map, etc. without using anonymous inner classes. If you use anonymous inner classes, you end up with a ratio of like 8:1 of boilerplate:relvant expressions. Believe me, I've tried it. If you want to see just how baroque it gets, take a look at functionaljava.org [functionaljava.org].

      Groovy

  • Proving that.. (Score:2, Insightful)

    by hexghost (444585)

    Twitter's developers care more about being cool and hip and using the latest tool so that they remain popular, than they do about having a site that stays up 7 days a week.

    • Re:Proving that.. (Score:4, Insightful)

      by AKAImBatman (238306) * <<akaimbatman> <at> <gmail.com>> on Tuesday April 07, 2009 @04:43PM (#27495367) Homepage Journal

      Twitter's developers care more about being cool and hip

      Not to be too pedantic, but doesn't that sum up Twitter as a whole?

      • Yeah, I mean, don't let them get too caught up with features or anything. Maybe some day, they'll add some features, but right now, they're just selling cool.
    • Re:Proving that.. (Score:4, Insightful)

      by joe_bruin (266648) on Tuesday April 07, 2009 @04:54PM (#27495545) Homepage Journal

      Twitter's developers care more about being cool and hip and using the latest tool so that they remain popular, than they do about having a site that stays up 7 days a week.

      Exactly. Scalability problems arise from poor implementation, not from language choices. Scalable platforms have been implemented in the past with PHP, ASP, Perl, C, Java, and I'm sure with Ruby, Python, or your favorite new language. Twitter is a massive-scale site, they should be looking at deep engineering, not a buzzword platform that promises easy scalability for dummies.

      Scala may help them alleviate problems they've hit in the Rails framework. What will help them with the problems they hit in Scala?

      • by Algan (20532)

        What will help them with the problems they hit in Scala?

        Erlang :)

        Seriously speaking, when you cannot have long lived processes due to memory leaks, you're having a language/platform problem. When you only have green threads, you're having a language/platform problem. Can you architect around it? Probably, but it may be sub-optimal, and bottom line is, why bother when there are better tools available?

      • Re:Proving that.. (Score:4, Informative)

        by burris (122191) on Tuesday April 07, 2009 @05:57PM (#27496425)

        According to the rebuttals in the comments of the blog post in one of my sibling posts here, part of Twitter's scalability problem was poor implementation of the Ruby interpreter. Lots of small objects cause the heap to get fragmented and eventually it runs out of memory. Java interpreters have better GC and you can swap out different GC algorithms in some of them.

        Why does everyone assume the people at Twitter are a bunch of newbies who don't know about deep engineering? Is it just because their analysis didn't lead them to your preferred buzzword?

  • Useful! (Score:5, Insightful)

    by castorvx (1424163) on Tuesday April 07, 2009 @04:42PM (#27495331)
    Twitter using new-and-fancy programming languages has a way of load testing them for all of us.

    I'm not sure I'd have the balls to take a 5 year old development platform/framework and drop it into something that sees so much traffic. Hopefully they share their experiences in some form.
  • by geekoid (135745) <dadinportland AT yahoo DOT com> on Tuesday April 07, 2009 @04:45PM (#27495403) Homepage Journal

    replace one language that wasn't tested on that scale and replace it with another one that wasn't tested on that scale.

    Good thinking~

    Oh look, twitter is down..again.

    • Re: (Score:3, Insightful)

      by AKAImBatman (238306) *

      As the joke I posted above insinuates, Scala runs on top of the Java platform. And unlike Ruby, it focuses on the use of the platform's features. So the platform is more than tested enough. Why they feel the need to use Scala rather than straight-up Java is one of life's great mysteries. But for now, their platform should be fine.

      Whether the code they write is scalable and holds up under loads or not is an entirely different topic.

      • Re:Good thinking, (Score:4, Interesting)

        by The Slashdolt (518657) on Tuesday April 07, 2009 @05:34PM (#27496129) Homepage
        Read this and all will become clear:
        Event-Based Programming without Inversion of Control [lamp.epfl.ch]
      • Re:Good thinking, (Score:5, Interesting)

        by burris (122191) on Tuesday April 07, 2009 @05:46PM (#27496293)

        Maybe they use Scala because writing Java code is painful by comparison. Tons of boilerplate, every exception has to be caught in every scope, no pattern matching, no named arguments, and on and on. For people like me, without Scala the JVM wouldn't even be under consideration, though I admit that Java has been more usable since it got generics.

  • by bogaboga (793279) on Tuesday April 07, 2009 @04:55PM (#27495553)

    Scala is not for me I can confidently say. I am too old to learn a new [programming] language. The languages I know will suffice for now.

    There is a saying too: "You cannot teach old dogs new tricks."

    • by Anonymous Coward on Tuesday April 07, 2009 @05:08PM (#27495763)
      You know, at first I thought this post was off-topic, because it's an incredibly pointless life update that none of us could possibly care about.

      Then I realized that the story is about Twitter, and suddenly I think it's the most relevant post so far.
  • by gpig (244284) on Tuesday April 07, 2009 @05:01PM (#27495651)

    Isn't she cute :)

  • Scala is great (Score:5, Interesting)

    by burris (122191) on Tuesday April 07, 2009 @05:04PM (#27495681)

    If I want to use any Java software then I'll use Scala. I see people bashing Scala, saying the languages they know are good enough or they can just use jython/jruby/groovy, but they clearly know little about Scala.

    One thing that's nice about Scala that Java, Jython, JRuby, and Groovy all lack is it's powerful type system and pattern matching. Once you get used to good pattern matching like in Scala, SML, OCaml, or Haskell you won't want to go back. Plus you get all the benefits of running on the JVM at high speed (unlike all the aforementioned JVM languages, except Java itself.)

    Honestly, you should check out Scala before you bash it. It's a very good choice wherever you might choose Java, which is a good choice for the back end. Twitter's developers are smart and experienced. They didn't choose Scala just to be cool. It is a powerful tool that can get the job done in an elegant way.

    • Why wouldn't Groovy be a suitable replacement? (Curious about Scala).

      • by burris (122191)

        Groovy is very slow [debian.org]. Maybe that will get better in Java 7. There are probably other meaningful differences that I would be able to point out if I were better versed in Groovy.

  • Mod down (Score:5, Funny)

    by Anonymous Coward on Tuesday April 07, 2009 @05:06PM (#27495721)

    OP is just a twitter sock puppet.

  • by toby (759)

    Java and Erlang did. Python isn't particularly 'functional' despite some recent syntactic grafts.

  • by tieTYT (989034) on Tuesday April 07, 2009 @05:18PM (#27495915)

    http://unlimitednovelty.com/2009/04/twitter-blaming-ruby-for-their-mistakes.html [unlimitednovelty.com]

    This blog post takes the attitude that Twitter didn't move to Scala because ROR had a problem, but because the in-house messaging system Twitter created performed poorly. The author does not work at Twitter but many of the Twitter developers (including Alex Payne) respond in the comments. I found the article to be very interesting and the comments even more so. They give a sense of how much research Twitter did before this change.

  • At first I read the title as "Twitter on Scalia". Justice Scalia is one of the most conservative US Supreme Court justices, and I'm not sure that this would have been a happy combination.
  • Psht. (Score:5, Funny)

    by kkrajewski (1459331) on Tuesday April 07, 2009 @09:04PM (#27498219) Journal

    I program in PDP-11 assembly, which is then translated into C, compiled into Java bytecode, and executed on a JVM. I call it Assemblacava, and it's the wave of the future.

  • scala vs erlang (Score:3, Informative)

    by xkcd150 (1527245) on Wednesday April 08, 2009 @02:17AM (#27500159)
    i figured a lot of people would mention erlang, and thought someone might be interested in this writeup i read the other day http://yarivsblog.com/articles/2008/05/18/erlang-vs-scala/ [yarivsblog.com]

Prototype designs always work. -- Don Vonada

Working...