Want to read Slashdot from your mobile device? Point it at m.slashdot.org and keep reading!


Forgot your password?
Programming Software IT Technology

Twitter On Scala 324

machaut writes "Twitter, one of the highest profile Ruby on Rails-backed websites on the Internet, has in the past year started replacing some of their Ruby infrastructure with an emerging language called Scala, developed by Martin Odersky at Switzerland's École Polytechnique Fédérale de Lausanne. Although they still prefer Ruby on Rails for user-facing web applications, Twitter's developers have started replacing Ruby daemon servers with Scala alternatives, and plan eventually to serve API requests, which comprise the majority of their traffic, with Scala instead of Ruby. This week several articles have appeared that discuss this shift at Twitter. A technical interview with three Twitter developers was published on Artima. One of those developers, Alex Payne, Twitter's API lead, gave a talk on this subject at the Web 2.0 Expo this week, which was covered by Technology Review and The Register."
This discussion has been archived. No new comments can be posted.

Twitter On Scala

Comments Filter:
  • by snowraver1 ( 1052510 ) on Tuesday April 07, 2009 @04:41PM (#27495323)
  • Proving that.. (Score:2, Insightful)

    by hexghost ( 444585 ) on Tuesday April 07, 2009 @04:42PM (#27495327) Homepage

    Twitter's developers care more about being cool and hip and using the latest tool so that they remain popular, than they do about having a site that stays up 7 days a week.

  • Useful! (Score:5, Insightful)

    by castorvx ( 1424163 ) on Tuesday April 07, 2009 @04:42PM (#27495331)
    Twitter using new-and-fancy programming languages has a way of load testing them for all of us.

    I'm not sure I'd have the balls to take a 5 year old development platform/framework and drop it into something that sees so much traffic. Hopefully they share their experiences in some form.
  • by Foofoobar ( 318279 ) on Tuesday April 07, 2009 @04:43PM (#27495365)
    Never was a nail except for the Ruby community that was in denial. Ever developer in the world except Ruby fanboys knew Ruby's inherent problem was scalability. Benchmarks showed it but they would always slant their own benchmarks to show the opposite. But facts are facts and in the end you can't deny the truth. So this is where we are at. The question is will Ruby fanboys still choose to deny the issues with Ruby or accept that it does have inherent limitations?
  • Re:Proving that.. (Score:4, Insightful)

    by AKAImBatman ( 238306 ) * <akaimbatmanNO@SPAMgmail.com> on Tuesday April 07, 2009 @04:43PM (#27495367) Homepage Journal

    Twitter's developers care more about being cool and hip

    Not to be too pedantic, but doesn't that sum up Twitter as a whole?

  • by geekoid ( 135745 ) <dadinportland AT yahoo DOT com> on Tuesday April 07, 2009 @04:47PM (#27495421) Homepage Journal

    It's new! It's hot! it's what all the kids are doing! It's what crappy programmers can pretend to adapt to instead of writing solid code!


  • by Anonymous Coward on Tuesday April 07, 2009 @04:47PM (#27495425)

    Scala compiles to Java Bytecode and runs on an unmodified Java Virtual Machine. It can call any Java Library and you can write code in Scala that is callable from Java.

    Scala offers much more than simple syntactic sugar. Take a look at their documentation. Especially the nice support for embedded DSLs is VERY interesting.

  • by AKAImBatman ( 238306 ) * <akaimbatmanNO@SPAMgmail.com> on Tuesday April 07, 2009 @04:50PM (#27495475) Homepage Journal

    As the joke I posted above insinuates, Scala runs on top of the Java platform. And unlike Ruby, it focuses on the use of the platform's features. So the platform is more than tested enough. Why they feel the need to use Scala rather than straight-up Java is one of life's great mysteries. But for now, their platform should be fine.

    Whether the code they write is scalable and holds up under loads or not is an entirely different topic.

  • by julesh ( 229690 ) on Tuesday April 07, 2009 @04:50PM (#27495481)

    Difference is, Facebook is still using php, Twitter is going toScala.

    PHP was a mature environment when facebook was launched. RoR was (and still is to a certain extent) a fad environment, popular primarily because of its differentness. People who build sites on a platform because it's the latest thing are less likely to stick with that platform than people who choose a platform that has a solid reputation but is boring. Scala, at a guess, is going to be the next fad platform. Like Ruby, it has some interesting ideas behind it, but it needs a lot of development before we can consider a stable platform for serious applications, I think.

  • by mini me ( 132455 ) on Tuesday April 07, 2009 @04:51PM (#27495491)

    While Facebook uses PHP where Twitter uses Rails, Facebook uses a plethora of languages to make the whole system work. So Twitter really isn't going to Scala any more than Facebook is going to Erlang. Which is the say that they use the best tool for the job, not one tool for every job.

  • by K. S. Kyosuke ( 729550 ) on Tuesday April 07, 2009 @04:52PM (#27495517)
    Ruby is a language. Languages usually don't have problems with scalability.
  • Re:Proving that.. (Score:4, Insightful)

    by joe_bruin ( 266648 ) on Tuesday April 07, 2009 @04:54PM (#27495545) Homepage Journal

    Twitter's developers care more about being cool and hip and using the latest tool so that they remain popular, than they do about having a site that stays up 7 days a week.

    Exactly. Scalability problems arise from poor implementation, not from language choices. Scalable platforms have been implemented in the past with PHP, ASP, Perl, C, Java, and I'm sure with Ruby, Python, or your favorite new language. Twitter is a massive-scale site, they should be looking at deep engineering, not a buzzword platform that promises easy scalability for dummies.

    Scala may help them alleviate problems they've hit in the Rails framework. What will help them with the problems they hit in Scala?

  • by hedleyroos ( 817147 ) on Tuesday April 07, 2009 @04:54PM (#27495547)

    I'm confused - how does a language "scale"? Can you suddenly have 1 billion items in your array instead of 100 million? If that is indeed the case then like the parent said: it is crappy coding and/or design.

    I'm fairly confident you can do twitter in whatever language you want and if designed properly it will scale in the proper sense of the word.

  • by A.K.A_Magnet ( 860822 ) on Tuesday April 07, 2009 @04:58PM (#27495615) Homepage
    I read between the lines that you call C or C++ solid-code, and if I'm not mistaken, you will find that the kids are doing Scala because the code is more solid. Scala benefits from a typing system close to OCaml's which makes Scala code very, very solid -- especially if you keep away Java specifics (such as nullable objects) in your code and take special care when interacting with Java libs that may do so.

    If I'm mistaken and you're not talking about C/C++, I hope you are not talking about dynamic languages which offer no guarantee whatsoever; you know as a developer I enjoy actually spending my time on working on the business side of my application -- and how to make it scalable, rather than working on low-level specifics and on testing if every pointer is null before dereferencing them. A type system that does this for me (which Scala or ML's parametrized type Option allows) is a bliss.

    Now, I'm not enumerate every language under the sun to see what code you call solid, I guess your answer would be that the code is solid whatever the language it's written in. In the end, it all comes down to binary instructions, right? The question is: how many guarantees do the tools give you? In the case of Scala's compiler, it gives you a lot AND offers you a very enjoyable, lightweight yet powerful syntax.
  • by nothing2seehere ( 1496253 ) on Tuesday April 07, 2009 @05:07PM (#27495739)
    Wow, Demi Moore? Really? Now that's an influential user!
  • by nicolas.kassis ( 875270 ) on Tuesday April 07, 2009 @05:08PM (#27495759)
    Except most languages have more than one interpreter/compiler whit varying performance in varying environments. Jython/IronPython/Python all perform a little different. Ruby 1.9 and MacRuby don't perform the same.
  • by tcopeland ( 32225 ) <tom@thomaslee[ ]eland.com ['cop' in gap]> on Tuesday April 07, 2009 @05:08PM (#27495761) Homepage

    > Ruby is a language. Languages usually don't have problems with scalability.

    Quite right. An application with 8 million users will have scalability challenges regardless of what type of language opcodes are being executed. At some point it's all about architecture.

  • by iluvcapra ( 782887 ) on Tuesday April 07, 2009 @05:43PM (#27496247)

    RoR was (and still is to a certain extent) a fad environment, popular primarily because of its differentness.

    Huh, I generally use it because it has really good ORM and migrations, and I really like the syntax (coming from Objective-C it's pretty slick. I also used the PHP language when I was starting out, but one day it tried to insist that $myArr[0] and $myArr["0"] actually pointed to the same object, and I have refused to deal with it ever since; I also got tired of typing str_sub_case_insensitive_for_real_safe(haystack, needle) -- or is it needle, haystack? And is this one of those prank functions that fails to substitute the value but still returns a value that evals true? Or if I leave out one of those underscores, am I in fact calling a function that behaves almost exactly the same way but fails under difficult-to-reproduce circumstances? Maybe they've fixed this and the other sundry atrocities? Maybe they've stopped trying to make it into Perl, as compiled by a C++ compiler, and tried fashioning it into an actual dynamic language? I know, I know, some people like PHP, but I think arguments for the superiority of PHP over Ruby (or Python or Scala or Lisp or WebObjects or Perl6 or really anything else) are going to rest completely on the skills of the Zend interpreter writers, and almost never on the quality/readability/maintainability of the code, or the ease of the development process. You can write good safe code in PHP, that is true, but it isn't very ergonomic.

    You know, RoR is really good at replacing those old Paradox and FMP database systems. I can see how Facebook might prefer PHP, but people trying to replace little inventory/business processes systems generally only need to support a few dozen users, and don't have an army of developers to keep it running. The Universe is big enough to accommodate the utility of Ruby on Rails and the Twitter developer's stupidity.

  • by nicolas.kassis ( 875270 ) on Tuesday April 07, 2009 @05:47PM (#27496297)

    Also Ruby is just slow.

    Not anymore. Go install 1.9 and start using it TODAY.

    The ruby community needs to leave 1.8.* to die as quickly as possible.

  • by K. S. Kyosuke ( 729550 ) on Tuesday April 07, 2009 @05:49PM (#27496341)

    Visual Basic was so stupid that Microsoft could have easily made it almost as fast as C - if only they had wanted. I guess they had other priorities.

    I must ask one thing: Have you ever heard of Self 93? They made it so good fifteen years ago that it topped at half the speed of C...a language where everything (!) was a message send and *anything* could have been redefined at *any* moment.

    If someone threw the necessary money at Ruby, it could have easily the performance of a modern Smalltalk implementation, as Ruby is essentially Smalltalk in diguise. The only major difference I spotted was the fact that Ruby objects behave as hashtables, whereas Smalltalk objects are essentially arrays of references with methods fetching the references ("instance variables") using fixed indices that change only during class redefinitions. I guess you could fake the former with the latter, enjoying a speedup, as even in Ruby most objects of a single class have the same attributes.

    But still, I guess that the perf. problems of Twitter are primarily due to Rails and not to Ruby.

  • by Foofoobar ( 318279 ) on Tuesday April 07, 2009 @06:15PM (#27496635)
    Listen to your own Ruby rhetoric. Ruby itself doesn't have threading; it's FAUX threads. And PHP was never meant to be used for app development; it's meant for web development. That would be like trying to build an OS in Visual Basic (now watch some idiot try).

    Yeah I understand what they are doing with Scala. And They DID try to do alot of it with Ruby over the years and Ruby people touted it over and over while we said it doesn't scale over and over and pointed out how they were removing more and more Ruby on the backend. Now you Ruby people say it was never on the backend? Unbelievable. You really do live in a world in yoru own mind. Hopefully you didn't build it on Ruby cause there's only room for one before it crashes.
  • Re:Scala is great (Score:2, Insightful)

    by listen ( 20464 ) on Tuesday April 07, 2009 @06:22PM (#27496701)

    Maybe because its a slow, me-too, mashed together dynamic language rather than a cohesive, statically typed language, designed in a peer reviewed environment with modularity, performance, safety, concurrency, and expressivity in mind?

  • by Jane Q. Public ( 1010737 ) on Tuesday April 07, 2009 @06:22PM (#27496707)
    As many programmers can tell you, publishing a successful book these days is worth a lot more than just the dollars it brings in directly; there is recognition and reputation, and perhaps a better-paying job down the road. Whether Payne is interested in the latter I do not know, but my point is simply that the book royalties are sometimes the least of the returns for being a successful author.
  • by maxume ( 22995 ) on Tuesday April 07, 2009 @06:44PM (#27496943)

    If you announce it to the world, it isn't 'hidden'. He may very well have an agenda, but he isn't being particularly disingenuous about it.

    And really, Twitter never said 'Ruby and Rails don't scale', they said 'This other stuff works better for what we are trying to do, for various reasons'.

  • by K. S. Kyosuke ( 729550 ) on Tuesday April 07, 2009 @07:31PM (#27497373)
    No, my logic is that Ruby is essentially a member of a well-researched class of languages, for which (the class, not Ruby) high-performance VMs have already been developed (Cincom Smalltalk, Gemstone, Self93, Strongtalk...), but this development was always expensive (this was also the case of Java and .NET, obviously). IMO, your shit(ty?) analogy does not apply here.
  • by DragonWriter ( 970822 ) on Tuesday April 07, 2009 @08:04PM (#27497687)

    Or is there any high level 'scripting' language out there that can actually measure up to a decent Lisp implementation?

    Since many decent Lisps are high-level dynamic languages (hence, "scripting" languages) clearly so.

  • by kv9 ( 697238 ) on Tuesday April 07, 2009 @08:24PM (#27497875) Homepage
    I don't get this buzzword-bingo bullshit about Twitter (or whatever the fuck site-du-jour is) in regards to concurrency and scalability. this is not a complex application, it's something that you code one afternoon (in Java/PHP) then throw it in a rack full of HTTP server nodes, a load balancer (shit, even RR-DNS will do) and a RAMSAN for the DB. that's it. stop the drama.
  • by Aloisius ( 1294796 ) on Tuesday April 07, 2009 @08:59PM (#27498167) Homepage

    And why do you need more than process-level concurrency (or currency at all?) for a system as simple as Twitter's? They aren't exactly doing heavy calculations.

  • by ClassMyAss ( 976281 ) on Wednesday April 08, 2009 @12:45AM (#27499751) Homepage
    Welcome to 1998.

    In the real world circa 2009, I routinely port C++ code to Java, where it runs almost as fast (usually within 20% or so) as the C++, sometimes even a bit faster if the algorithms are not very cache-friendly.

    The JVM is very fast, especially if you're using it in a production environment with the right flags set.
  • by iluvcapra ( 782887 ) on Wednesday April 08, 2009 @01:52AM (#27500075)

    The fact that $arr[0] is the same as $arr['0'] is just insane

    I'm not even sure if that was what it was, I think the problem I was having on top of that was that I would pop something off the head of $arr and sometimes it would be $arr[0] and sometimes it would be $arr[1], and this is crazy, but I dug in and low and behold I was actually storing values in the array to ["0"] and ["1"], and since arrays and hashes are the same thing in PHP (this is the real sin), it did what it thought was right, and created a non-deterministic bug. Why does it let you pop a value off a hash? UGH!

    But one day I discovered Ruby, and the first lesson I learned was that error messages are actually a good thing, instead of the PHP developers practice, wherein they construct the entire language around the principle of guessing what you mean and never reporting errors. Until the interpreter segfaults.

  • by kv9 ( 697238 ) on Wednesday April 08, 2009 @11:08AM (#27503785) Homepage

    That's fine for the vast majority of web applications, but you clearly don't understand the scale of the traffic a site like Twitter receives.

    I clearly don't. this type of webapp generally scales well, their problem is that they chose shitty technologies to "power" it. you want fast? code an Apache module that uses libmysql (or whatever DB it is they use). now that's fast.

    What do you do when one database machine, no matter how fast, isn't enough?

    I believe they can split the DB, add more RAM, cache a lot (a LOT, cache it in RAM, RAM is cheap), serve static content.

    When your load balancer gets overloaded?

    in this scenario the load balancer is the least of your problems. what happens if a switch dies? drives die, PSUs blow up, etc etc. use redundancy.

    How about handling that massive search index?

    I don't know about that, let me google it.

User hostile.