Twitter On Scala 324
machaut writes "Twitter, one of the highest profile Ruby on Rails-backed websites on the Internet, has in the past year started replacing some of their Ruby infrastructure with an emerging language called Scala, developed by Martin Odersky at Switzerland's École Polytechnique Fédérale de Lausanne. Although they still prefer Ruby on Rails for user-facing web applications, Twitter's developers have started replacing Ruby daemon servers with Scala alternatives, and plan eventually to serve API requests, which comprise the majority of their traffic, with Scala instead of Ruby. This week several articles have appeared that discuss this shift at Twitter. A technical interview with three Twitter developers was published on Artima. One of those developers, Alex Payne, Twitter's API lead, gave a talk on this subject at the Web 2.0 Expo this week, which was covered by Technology Review and The Register."
Who gives a shit about twitter? (Score:5, Insightful)
Proving that.. (Score:2, Insightful)
Twitter's developers care more about being cool and hip and using the latest tool so that they remain popular, than they do about having a site that stays up 7 days a week.
Useful! (Score:5, Insightful)
I'm not sure I'd have the balls to take a 5 year old development platform/framework and drop it into something that sees so much traffic. Hopefully they share their experiences in some form.
Re:Should have used PHP. (Score:3, Insightful)
Re:Proving that.. (Score:4, Insightful)
Not to be too pedantic, but doesn't that sum up Twitter as a whole?
Re:Scala seems to be Java+/- (Score:4, Insightful)
It's new! It's hot! it's what all the kids are doing! It's what crappy programmers can pretend to adapt to instead of writing solid code!
weeee!!!
Re:Scala seems to be Java+/- (Score:2, Insightful)
Scala compiles to Java Bytecode and runs on an unmodified Java Virtual Machine. It can call any Java Library and you can write code in Scala that is callable from Java.
Scala offers much more than simple syntactic sugar. Take a look at their documentation. Especially the nice support for embedded DSLs is VERY interesting.
Re:Good thinking, (Score:3, Insightful)
As the joke I posted above insinuates, Scala runs on top of the Java platform. And unlike Ruby, it focuses on the use of the platform's features. So the platform is more than tested enough. Why they feel the need to use Scala rather than straight-up Java is one of life's great mysteries. But for now, their platform should be fine.
Whether the code they write is scalable and holds up under loads or not is an entirely different topic.
Re:Should have used PHP. (Score:5, Insightful)
Difference is, Facebook is still using php, Twitter is going toScala.
PHP was a mature environment when facebook was launched. RoR was (and still is to a certain extent) a fad environment, popular primarily because of its differentness. People who build sites on a platform because it's the latest thing are less likely to stick with that platform than people who choose a platform that has a solid reputation but is boring. Scala, at a guess, is going to be the next fad platform. Like Ruby, it has some interesting ideas behind it, but it needs a lot of development before we can consider a stable platform for serious applications, I think.
Re:Should have used PHP. (Score:5, Insightful)
While Facebook uses PHP where Twitter uses Rails, Facebook uses a plethora of languages to make the whole system work. So Twitter really isn't going to Scala any more than Facebook is going to Erlang. Which is the say that they use the best tool for the job, not one tool for every job.
Re:Should have used PHP. (Score:5, Insightful)
Re:Proving that.. (Score:4, Insightful)
Twitter's developers care more about being cool and hip and using the latest tool so that they remain popular, than they do about having a site that stays up 7 days a week.
Exactly. Scalability problems arise from poor implementation, not from language choices. Scalable platforms have been implemented in the past with PHP, ASP, Perl, C, Java, and I'm sure with Ruby, Python, or your favorite new language. Twitter is a massive-scale site, they should be looking at deep engineering, not a buzzword platform that promises easy scalability for dummies.
Scala may help them alleviate problems they've hit in the Rails framework. What will help them with the problems they hit in Scala?
Re:Scala seems to be Java+/- (Score:2, Insightful)
I'm confused - how does a language "scale"? Can you suddenly have 1 billion items in your array instead of 100 million? If that is indeed the case then like the parent said: it is crappy coding and/or design.
I'm fairly confident you can do twitter in whatever language you want and if designed properly it will scale in the proper sense of the word.
Re:Scala seems to be Java+/- (Score:5, Insightful)
If I'm mistaken and you're not talking about C/C++, I hope you are not talking about dynamic languages which offer no guarantee whatsoever; you know as a developer I enjoy actually spending my time on working on the business side of my application -- and how to make it scalable, rather than working on low-level specifics and on testing if every pointer is null before dereferencing them. A type system that does this for me (which Scala or ML's parametrized type Option allows) is a bliss.
Now, I'm not enumerate every language under the sun to see what code you call solid, I guess your answer would be that the code is solid whatever the language it's written in. In the end, it all comes down to binary instructions, right? The question is: how many guarantees do the tools give you? In the case of Scala's compiler, it gives you a lot AND offers you a very enjoyable, lightweight yet powerful syntax.
Re:Who gives a shit about twitter? (Score:2, Insightful)
Re:Should have used PHP. (Score:3, Insightful)
Re:Should have used PHP. (Score:5, Insightful)
> Ruby is a language. Languages usually don't have problems with scalability.
Quite right. An application with 8 million users will have scalability challenges regardless of what type of language opcodes are being executed. At some point it's all about architecture.
Re:Should have used PHP. (Score:5, Insightful)
RoR was (and still is to a certain extent) a fad environment, popular primarily because of its differentness.
Huh, I generally use it because it has really good ORM and migrations, and I really like the syntax (coming from Objective-C it's pretty slick. I also used the PHP language when I was starting out, but one day it tried to insist that $myArr[0] and $myArr["0"] actually pointed to the same object, and I have refused to deal with it ever since; I also got tired of typing str_sub_case_insensitive_for_real_safe(haystack, needle) -- or is it needle, haystack? And is this one of those prank functions that fails to substitute the value but still returns a value that evals true? Or if I leave out one of those underscores, am I in fact calling a function that behaves almost exactly the same way but fails under difficult-to-reproduce circumstances? Maybe they've fixed this and the other sundry atrocities? Maybe they've stopped trying to make it into Perl, as compiled by a C++ compiler, and tried fashioning it into an actual dynamic language? I know, I know, some people like PHP, but I think arguments for the superiority of PHP over Ruby (or Python or Scala or Lisp or WebObjects or Perl6 or really anything else) are going to rest completely on the skills of the Zend interpreter writers, and almost never on the quality/readability/maintainability of the code, or the ease of the development process. You can write good safe code in PHP, that is true, but it isn't very ergonomic.
You know, RoR is really good at replacing those old Paradox and FMP database systems. I can see how Facebook might prefer PHP, but people trying to replace little inventory/business processes systems generally only need to support a few dozen users, and don't have an army of developers to keep it running. The Universe is big enough to accommodate the utility of Ruby on Rails and the Twitter developer's stupidity.
Re:Should have used PHP. (Score:3, Insightful)
Also Ruby is just slow.
Not anymore. Go install 1.9 and start using it TODAY.
The ruby community needs to leave 1.8.* to die as quickly as possible.
Re:Should have used PHP. (Score:3, Insightful)
Visual Basic was so stupid that Microsoft could have easily made it almost as fast as C - if only they had wanted. I guess they had other priorities.
I must ask one thing: Have you ever heard of Self 93? They made it so good fifteen years ago that it topped at half the speed of C...a language where everything (!) was a message send and *anything* could have been redefined at *any* moment.
If someone threw the necessary money at Ruby, it could have easily the performance of a modern Smalltalk implementation, as Ruby is essentially Smalltalk in diguise. The only major difference I spotted was the fact that Ruby objects behave as hashtables, whereas Smalltalk objects are essentially arrays of references with methods fetching the references ("instance variables") using fixed indices that change only during class redefinitions. I guess you could fake the former with the latter, enjoying a speedup, as even in Ruby most objects of a single class have the same attributes.
But still, I guess that the perf. problems of Twitter are primarily due to Rails and not to Ruby.
Re:Should have used PHP. (Score:3, Insightful)
Yeah I understand what they are doing with Scala. And They DID try to do alot of it with Ruby over the years and Ruby people touted it over and over while we said it doesn't scale over and over and pointed out how they were removing more and more Ruby on the backend. Now you Ruby people say it was never on the backend? Unbelievable. You really do live in a world in yoru own mind. Hopefully you didn't build it on Ruby cause there's only room for one before it crashes.
Re:Scala is great (Score:2, Insightful)
Maybe because its a slow, me-too, mashed together dynamic language rather than a cohesive, statically typed language, designed in a peer reviewed environment with modularity, performance, safety, concurrency, and expressivity in mind?
Re:There you go again! (Score:3, Insightful)
Re:There you go again! (Score:2, Insightful)
If you announce it to the world, it isn't 'hidden'. He may very well have an agenda, but he isn't being particularly disingenuous about it.
And really, Twitter never said 'Ruby and Rails don't scale', they said 'This other stuff works better for what we are trying to do, for various reasons'.
Re:Should have used PHP. (Score:4, Insightful)
Re:Should have used PHP. (Score:3, Insightful)
Since many decent Lisps are high-level dynamic languages (hence, "scripting" languages) clearly so.
Re:Should have used PHP. (Score:4, Insightful)
Re:Good thinking, (Score:2, Insightful)
And why do you need more than process-level concurrency (or currency at all?) for a system as simple as Twitter's? They aren't exactly doing heavy calculations.
Re:Aw, Java and Python had a baby! (Score:3, Insightful)
In the real world circa 2009, I routinely port C++ code to Java, where it runs almost as fast (usually within 20% or so) as the C++, sometimes even a bit faster if the algorithms are not very cache-friendly.
The JVM is very fast, especially if you're using it in a production environment with the right flags set.
Re:Should have used PHP. (Score:3, Insightful)
I'm not even sure if that was what it was, I think the problem I was having on top of that was that I would pop something off the head of $arr and sometimes it would be $arr[0] and sometimes it would be $arr[1], and this is crazy, but I dug in and low and behold I was actually storing values in the array to ["0"] and ["1"], and since arrays and hashes are the same thing in PHP (this is the real sin), it did what it thought was right, and created a non-deterministic bug. Why does it let you pop a value off a hash? UGH!
But one day I discovered Ruby, and the first lesson I learned was that error messages are actually a good thing, instead of the PHP developers practice, wherein they construct the entire language around the principle of guessing what you mean and never reporting errors. Until the interpreter segfaults.
Re:Should have used PHP. (Score:3, Insightful)
That's fine for the vast majority of web applications, but you clearly don't understand the scale of the traffic a site like Twitter receives.
I clearly don't. this type of webapp generally scales well, their problem is that they chose shitty technologies to "power" it. you want fast? code an Apache module that uses libmysql (or whatever DB it is they use). now that's fast.
What do you do when one database machine, no matter how fast, isn't enough?
I believe they can split the DB, add more RAM, cache a lot (a LOT, cache it in RAM, RAM is cheap), serve static content.
When your load balancer gets overloaded?
in this scenario the load balancer is the least of your problems. what happens if a switch dies? drives die, PSUs blow up, etc etc. use redundancy.
How about handling that massive search index?
I don't know about that, let me google it.