How Twitter Is Moving To the Cassandra Database 157
MyNoSQL has up an interview with Ryan King on how Twitter is transitioning to the Cassandra database. Here's some detailed background on Cassandra, which aims to "bring together Dynamo's fully distributed design and Bigtable's ColumnFamily-based data model." Before settling on Cassandra, the Twitter team looked into: "...HBase, Voldemort, MongoDB, MemcacheDB, Redis, Cassandra, HyperTable, and probably some others I'm forgetting. ... We're currently moving our largest (and most painful to maintain) table — the statuses table, which contains all tweets and retweets. ... Some side notes here about importing. We were originally trying to use the BinaryMemtable interface, but we actually found it to be too fast — it would saturate the backplane of our network. We've switched back to using the Thrift interface for bulk loading (and we still have to throttle it). The whole process takes about a week now. With infinite network bandwidth we could do it in about 7 hours on our current cluster." Relatedly, an anonymous reader notes that the upcoming NoSQL Live conference, which will take place in Boston March 11th, has announced their lineup of speakers and panelists including Ryan King and folks from LinkedIn, StumbleUpon, and Rackspace.
Don't believe them! (Score:5, Funny)
heh heh heh.
Re: (Score:1)
Do you have an ex-girlfriend called Cassandra, by any chance?
Re: (Score:2)
Drink a few beers. Read the Iliad. You'll feel better.
(AJAX: When second-best is good enough. Or maybe AJAX is almost as good as ACHILLES.)
Re: (Score:1)
Re: (Score:2)
Damn... you beat me to it. I was going to say, "Cassandra? I don't believe it!"
Re: (Score:2)
For some reason my mind went to Cassandra Crossing (http://en.wikipedia.org/wiki/The_Cassandra_Crossing)
Cassandra, eh? (Score:5, Funny)
Re: (Score:3, Funny)
Especially when trojan horses are the cause of such a disaster....
Re: (Score:2)
Re: (Score:2)
Of course!
Because Cassandra is a Trojan....
Re: (Score:2)
Disastrous failure? Twitter? There’s at least one joke in there somewhere. ^^
network issues? (Score:5, Insightful)
.
First time I have ever heard anyone say that a database was too fast. Maybe there are network problems that also need to be addressed.
Re: (Score:2)
Could someone explain to me why this kind of speed would be a problem? It seems to me that if BinaryMemtable is so incredibly fast that other things become a bottleneck, then you're in a great position. You have something very fast for storing and retrieving data - you just need to get bigger, faster pipes.
Re: (Score:3, Insightful)
I know next to nothing about NoSQL, but what they're talking about there seems to be using BinaryMemtable for the one-time move of data. You can see that you wouldn't want to "saturate the backplane of our network" for several days while that completes, so they're using a slower method & throttling it. It will take a week to do the move, but everything else will keep working.
Re: (Score:2)
I'm surprised they didn't use the faster method and throtlle it.
Re:network issues? (Score:4, Informative)
Re: (Score:2)
That actually makes a lot of sense. Thanks!
Re: (Score:2)
Re: (Score:3, Informative)
Re: (Score:2)
Ah, that makes sense. For some reason I thought they were talking about general usage. Thanks for clearing that up. (o:
Re: (Score:2)
I haven't checked, but I'd bet that BinaryMemtable uses UDP, when combined with the fast speed, could easily cause significant network saturation..
Huzzzah! (Score:2)
I look forward to a brand new twitter that randomly doesn't display expected data and sometimes doesn't take my status updates!
And this is front page news, why? (Score:3, Interesting)
Why is it that whenever twitter makes any random change to some part of its infrastructure that we need a front page story about it?
Re:And this is front page news, why? (Score:5, Funny)
Why is it that whenever twitter makes any random change to some part of its infrastructure that we need a front page story about it?
Because the change prevented them from posting it to twitter.
Re:And this is front page news, why? (Score:5, Insightful)
Scaling. If something turns out to be robust and fast enough for Twitter, it is definitely of interest to anyone working on significantly large and busy websites.
Re:And this is front page news, why? (Score:4, Insightful)
Yes, because twitter is the epitome of robustness and speed. Oh wait... Just in the 2 months of this year alone they've had something like 4 outages.
Re:And this is front page news, why? (Score:5, Insightful)
No way. Their architecture is about as "best guess" engineering as Facebook. I don't think that's actually what engineering is. "Maybe this one will work?"
In the meantime, I have not been able to update my avatar image on Twitter, and TwitPic-like feature is still a faint glimmer in Twitter's amateur eyes. Speaking of missed opportunities, why drive so much traffic to Twitter parasites Bit.ly, TwitPic, TinyURL, Twitition, TwitLonger?
What in the world are Twitter's engineers actually DOING should be the real question.
Re: (Score:2)
That may not be what actual engineering is but that describes a lot of software "engineering"
Re: (Score:2)
Which is exactly why developers need to pay attention - So we can avoid these mistakes ourselves.
Re: (Score:2)
Well that's actually why I like this news.
I like to think of Twitter's technology experiments, as high not to build and run a high performance web application. Hell, they bought us confirmation that Ruby on Rails wasn't exactly ready for prime time in terms of high performance work for example.
We have a lot to thank them for, but you're right, one of those things is not how to run a stable, secure, scalable web site, it is the opposite- how not to. I suspect before long we'll be able to see for ourselves ho
Re: (Score:2)
So instead of just seeing what an actual robust and fast site uses we should instead follow Twitter which switches at whim to whatever is the technology de jour of the moment while still being unstable and slow?
Re: (Score:2)
Thanks for reminding me why I stopped coming here years ago.
Are you the Ghost of Slashdot Past per chance?
Re: (Score:2)
Re: (Score:3, Interesting)
Re: (Score:2)
Twwweeeeter can also probably generate static pages just as well on some large node and then push them to web servers, that just might have worked better for them.
Do they really need dynamic pages at all or could they live with something that's regenerated every 10 minutes? Just saying.
Re: (Score:2)
(1) In some measures , probably;
(2) When Google or Wikipedia makes announcements about technology (whether its a "change" or not) they use in their backend, that's usually often a front-page story on Slashdot, too. The BBC and CNN don't, AFAIK, tend to make big public announcements about back-end technology.
Re: (Score:3, Insightful)
I suppose then why would we care if any site made any random change to any part of its infrastructure?
Twitter is a -very- busy site.
They are changing their infrastructure to accomodate. Here's what they looked at, here is what they chose. If you are looking for something with equal performance, you don't have to shop around.
Re: (Score:2)
Because in some areas Twitter is at an extreme of scale, so what they are doing to deal with that extreme of scale (even if it isn't necessarily always the ideal choice) is usually interesting since, if you are looking for things that have been done in production to deal with the kind of scaling they experience, there aren't a lot of other data points to find.
pfffft twatter tweeter (Score:2, Insightful)
who cares what twuufter is running off.
The more interesting aspect of all of this 'NoSQL' movement is how they believe that if they achieve some speed improvement against some relational databases, how that makes them so much better.
If you don't really need a database to run your 'website', then who cares if you use flat files or an in memory hashmap for all your data needs? Databases are not being replaced by NoSQL in projects that need databases. The projects that may not have ever needed databases may b
Re: (Score:2)
Is there really a huge issue with rdbms speeds? I don't know perhaps you should pose that question to google for instance.
Re: (Score:3, Insightful)
your question is answered in my post: google does not need a database for ACID properties.
Can you complain much if in one location google gives you results that are very different for the same search query as for the same query in a different location at the same time? Well, if you do complain, you can ask google for your money back.
Re: (Score:2)
BigTable itself leverages map-reduce to cascade the query to potentially thousands of machines, reducing their results back to a SINGLE requesting node.
Geo-location would pick one of several data-centers which house an isolated effective database. The upper layered code would act identically
Re:pfffft twatter tweeter (Score:5, Insightful)
I think their point is not everything needs an RDBMS, whereas before it was the 'go to' method of storing data.
Re:pfffft twatter tweeter (Score:5, Insightful)
Or: use the right tool for the job. The only difference is, now alternative tools actually exist.
Re: (Score:2)
Re: (Score:2)
Yeah, no kidding, it's called a filesystem. But when was the last time you heard announced a mainstream, high-performance, non-relational data store that was intended to be an alternative to an RDBMS (BTW, I'm intentionally discounting OODBMSes, as I think they and RDBMSes are intended to target largely the same application space)? I know I haven't. People simply rolled their own and moved on. But times are changing and that niche is finally being filled (in part because that niche isn't so niche anymor
Re: (Score:2)
One of the most recent, well-known major suc
Re: (Score:2)
One of the most recent, well-known major successes before the recent "NoSQL" movement, in terms of a product that sacrificed ACID for performance as an alternative to databases providing ACID guarantees, was MySQL.
I said nothing about ACID compliance. I specifically mentioned non-relational datastores, and clearly MySQL isn't that. As such, it still forces the developer to work with a relational data model, and one of the main things these so-called "NoSQL" projects do is lift that requirement.
Aside from
Re: (Score:2)
Um, the reason MySQL with MyISAM doesn't provide ACID guarantees (particularly, its deficiencies with regard to consistency) are related to the ways in which MySQL with MyISAM fails to implement the relational model. Merely using a dialect of SQL as a query language doesn't make a database relational.
Re: (Score:2)
Just because you haven't heard of something doesn't make it obscure. Tens of millions of Americans still can't find Iraq on a map - doesn't mean it doesn't exist or isn't "mainstream".
And, at least one major "mainstream" US news network can tell Egypt from Iraq.
http://mediamatters.org/mmtv/200907270040 [mediamatters.org]
Caché / MUMPS is heavily used in Healthcare and Finance.
Your life and your financial future may well depend on apps that run on them.
Just because something isn't incessantly hyped by egocentric CE
Re: (Score:2, Insightful)
You know, the truth is, most data is still stored in individual files, not in databases. So RDBMSs were always a very niche thing used for projects because they are understood and it's easier to develop for them if you really have massive data requirements.
Files - that's what many projects even today use, not databases. This is basically what they are going back to - files with whatever window dressing on top - a facade of hashes, it's all key/value pairs. It is, my friends, the old old idea of property
Re: (Score:2)
Re:pfffft twatter tweeter (Score:4, Interesting)
I think you're missing the point here, the problem with RDBMSs isn't that they are "slow" per-se, which implies that they just need some good ol' fashioned optimization. The problem is that there is a cost associated with the data integrity guarantees they make (usually appears in scalability bottlenecks rather than in pure computational inefficiencies), regardless of how good the implementation is, and if you don't need some of those guarantees, you can dispense with them and end up with better performance (again, this typically means better scalability). Additionally, this is the kind of bottleneck that you just can't throw more resources at. Sure you can find the bottleneck and beef up that particular component to do more transactions/second, but at a certain point you've isolated the bottleneck on a world-class server that is doing nothing but that, and it's still a bottleneck. At that point (preferably long before you reach that point) you have to look at transitioning to an infrastructure that makes some kind of tradeoff that allows the removal of the bottleneck, which is what NoSQL does.
I doubt Twitter wants very many RDBMS-type data coherency guarantees at all. 160-character text strings with a similarly-sized amount of metadata, and no real-time delivery guarantees? Sounds like their database can get pretty inconsistent without messing things up badly. It seems to me they would be well served by using a database that offers just what they want/need in that area and better performance.
Oh and this:
Is there really a huge issue with rdbms speeds?
yes, and what are you smoking that you would even ask this question?
Re: (Score:2)
As I said, there are projects and then there are projects. Tweater is not the project that requires any real database in the first place, who cares is a commit is transactional there?
As for your last comment: problems with database performance are all about design. You think NoSQL will not hit the same roadblocks in projects that don't do design right? What are they going to move to when that one fails? NoNoSQL++?
Re: (Score:2)
I am going to add a few other things here. The first is that "not possible to scale" is not really accurate. I believe there are ways to design structures so that write capacity on an RDBMS can scale upward with the nodes on the network. Of course this only works for some types of applications (the approach I have in mind would work with Twitter, for example). And even with Amazon, you would CERTAINLY want RI on purchases even if you don't care about reviews.
However, the larger point is that an RDBMS is
Speed *isn't* scalability (Score:2)
Speed is latency. (how long it takes)
Scalability is throughput. (how many concurrent). Or put another way; Speed is the quality, throughput is the width.
who cares what twuufter is running off.
Well, developers, and their managers do. They're nothing if not fashion victims.
RDBMS aren't the be all and end all of scalability (or speed, they perform a shit load of management functions you may or may not need). While attempting to scale conventional rdbms you get into write consistency problem, lookup performance problems unless you specifically desig
Re: (Score:2)
I don't think you understand the niche that NoSQL databases are trying to fill.
It's not a black and white, panacea-type situation. Relational databases are good at some things, non-relational databases are good at others. Where non-relational databases are better is at solving very specific problems, many of which
Re: (Score:2)
Freedom of choice, definitely. I had projects just recently I used property files as a database - inserts, deletes, updates, all in a property file. Easy enough because it is just a hash map. You don't impress me with any of it, it's not in any way new first of all, but it does not replace any RDBMS where RDBMS is needed.
My entire point is that Twooter never needed an RDBMS in the first place. They should be just fine without any database usage on the front end, and forget about JSON. The problem with
Re: (Score:2)
What makes you think this has not been done? Sometimes combination of arrogance and ignorance here is amazing. Very bright minds are working on all kinds of approaches; and of course Oracle (et al) are working on their set of tools to improve them as well.
In reality it is ALL about different
Re: (Score:2)
Just like there is no universal programming language for every type of software, there is no universal database engine for every type of data storage.
Re: (Score:2)
Relatively expensive 15,00
Re: (Score:2)
Thank you for the informative and thought-provoking post. It's certainly refreshing to see discourse on a level above "I hate MySQL, therefore SQL sucks." You make some good points.
Nevertheless, an RDBMS is still the way to go. You hint at the reason in your last paragraph, actually. The entire NoSQL "movement" is predicated on a confusion of implementation and interface. You describe various problems with the way conventional RDBMSes employ the disk: who said RDBMSes had to use those approaches?
There's not
Re: (Score:2)
The problem is that with this kind of backing storage, you can't implement most of sql effective. So you might end up with a 'sql' database where you can't user joins in production due to performance. So you end up with the worst of both worlds. A 'relational' database where you can't use most of the relational operations due to performance issues. And you still have a relative interface, so you can't do the kind of magic optimizations you can do with a simple key/value storage.
As i see it, the problem is t
Re: (Score:2)
I've often found that DRBD+NFS flat file
Re: (Score:2)
There is a difference between needing a structured storage mechanism (database) and needing a database that implements the relational model and provides ACID guarantees. Further, many non-relational databases provide specific, weaker forms of ACID guarantees that are better than (say) naive flat file storage would, while providing better scalability in certain appli
he he he, hit the soft spot, did I, twater? (Score:2)
http://slashdot.org/~roman_mir/comments - I imagine twater storm of moderation points was spent well this time, every single post I had on this issue was above 3 point and now within 1 hour, all comments were moderated down. To me that's just funny - someone does not like the truth.
I just wonder is it the twater birds or does it have something to do with the nosql ideologists?
Re: (Score:2)
you really should have left that comment to the 'bad analogy guy', he could make it sound good.
If other tools are faster and better than rdbms, then why people should waste their time with slower option.
- faster and better, ha? So you don't really mind if your bank switches from its datastore to a 'faster and better' NoSQL system, whatever the latest fad name is? I mean what's a few dollars not rolling back in a transaction that fails when your employment check is deposited?
Propeller would have been a file in a makeshift file system, we use jet engines now for large commercial aircraft now, you don't see the r
Re: (Score:2)
you don't think I know what I mean? [answers.com]
Re: (Score:2)
goodness, that is a terrible article you are referring to, just awful.
Additionally, managing relational databases in a production environment can become labor intensive and error-prone. Each database package comes with its own world of configuration options, performance sensitivities, bugs, and tools
- just like any other software. What, the NoSQL software will not have its own 'labor intensive, error-prone, configuration options, performance sensitive, bugs and tools' features? What, they invented a way to write software in the NoSQL camp that can avoid being complex to maintain in real production environments while being totally bug free while not having complex configuration options and will not be performance sensitive?
While these issues usually start small, they can become a drain on developers' time and resources as the product matures and its needs become more complex. This complexity of management arises from the complexity of the database packages themselves; it is their very breadth of capabilities which makes them difficult to manage.
- and
Intersystems Caché (Score:2)
Re: (Score:2)
Not cheap, though.
That might be part of it....
Re: (Score:2)
Don't want to install Cassandra (Score:3, Funny)
I hear Cassandra is really a trojan. Can anyone verify? I don't want a trojan on my computer.....
Re: (Score:2)
I hear Cassandra is really a trojan. Can anyone verify? I don't want a trojan on my "computer....."
But, but... what if I gift it you? I swear I'm not Trojan but Greek.
Twitter needs scalability experts (Score:5, Interesting)
I love how ass backwards twitter has always been with learning how to scale their 90s infrastructure up. I remember when they called out the Ruby community because they didn't understand MySQL replication and memcached.
I guess without a profit model they couldn't use a real RDBMS like Oracle. EFD (Enterprise Flash Drive) support anyone? 11g supports EFD on native SSD block-levels. Write scale? How about 1+ million transactions/sec on a single node Oracle DB using <$100K worth of equipment and licenses? Anyway, I've built HUGE databases for a long time, odds are most of you have interfaced with them. Just because it's free and open-source doesn't make it cheap.
I love FOSS don't get me wrong, but best-in-class is best-in-class. I only use FOSS when it happens to be best-in-class. I laugh at how none of the requirements included disaster recovery. No single point of failure does not preclude failing at every point simultaneously. EMP bomb at your primary datacenter anyone?
Re: (Score:2)
Holy fuck, the right tool for the right job, please? Oracle does somethings for some markets really well but for the rest of us who don't need such a high degree of transactional safety that $90k + two-node RAC price tag might just end up taking your great web 3.0 business through development, maybe early beta before you begin liquidating assets. That's per-processor licensing too on a database that scales vertically well (very well really) but not horizontally well (sharding a
Re: (Score:2)
Sure, you could run Oracle on x86 desktop hardware. You could even go so far as buying two cheap e-machines and run them both as Linux-based RAC nodes. Granted, those shitty nodes with the fibre-channel attached disk array would still have set you back probably over $90k with the additional hardware and licenses.
Eventually, if you're lucky enough to see rapid or continued growth you'll find yourself moving onto better enterprise gra
Re: (Score:2)
They are seeing about 1/2 million transactions per second with this setup based on the information given, but no word of what their cluster consists of. If it is just a handful of generic PCs, $100,000 for your setup looks pretty expensive.
Re: (Score:2, Informative)
Re: (Score:2)
I love oracle it is a fine database, would I personally buy it? Nope, but as long as
it is OPM (Other Peoples Money) I am perfectly fine with it. Now say I was designing something
like a medical records system oracle would be a no brainer. Missing a couple of tweets here
and there who is really going to care.
Re: (Score:2)
I only use FOSS when it happens to be best-in-class
Just curious, what FOSS have/do you use?
Re: (Score:2)
I am curious what someone with your experience thinks of PostgreSQL ? Would you say that it can scale properly as Oracle does ?
This is a genuine question as I am pondering between both for my startup. Even thought I already done my investigations, one more opinion cannot hurt :) Assuming my current DB design holds, it will have about 50 tables, most having less than 10,000 records and some having few millions records (they will be partitioned). The volume of reads will be much higher than writes. Write quer
Re: (Score:2)
1) They never said they didn't plan for disaster recovery. It's silly to deride them for not discussing the entirety of their backups and disaster recovery efforts when the whole topic of the article was their move to Cassandra as a primary data store.
2) Disaster recovery looks at realistic threat scenarios. Fire, s
Re: (Score:2, Insightful)
I've never dealt with an EMP but a more realistic threat with similar effects would be planning for a hurricane or earthquake. I used to work at a international bank and we had to deal with both (offices in FL and CA). For the most part the best solution was to have an identical setup at another office and having all applications available via VPN and/or web access. We had a separate pipe that was used only for backup data transfers. The DB transaction logs were written both locally and remotely. All u
Re: (Score:2)
you are a more hardened sysadmin than us all.
You have no idea...
http://xkcd.com/705/
Java / JVM Wins Again ... (Score:2)
It's fascinating how after initially being a posterboy for the post-Java revolution Twitter is gradually moving their architecture to the JVM, piece by piece. I think it's actually a credit to them that they seem to have level heads and are evaluating technology on it's merits (where as if you talk to most of the ruby / python crowd they would rather stick toothpicks in their eyes than endorse a solution that involves java).
Re: (Score:3, Funny)
Until recently I thought the same way, I would never endorse a solution that involves java. However
a recently came to the same realization that sun did when they created it. Java is a fantastic
way to over sell gobs of expensive hardware. I am a system administrator so the more hardware it takes to
run a solution the better off I am, more machines, more money and better job security. So I have now
fully jumped on the java bandwagon, java makes me smile.
Re: (Score:3, Informative)
Sure - but I think the whole point is that you'd be smiling even more if they were using one of the modern & trendy dynamic languages because you'd likely have 2 - 3 times the amount of hardware to look after. I'm not sure what alternative you would propose that uses less hardware but there actually aren't many that are better than the JVM these days.
Re: (Score:2)
I think its fascinating, too -- but probably in a very different way than you do. You seem to think that it is a repudiation of some mythical "post-Java revolution", when in many ways I think it is a validation of exactly the approach that was common to pushing Ruby, Python, and similar languages as more agile alternatives to Java. The appeal of tools n
Open Source Parallel Databases (Score:2)
Re: (Score:2)
I have actually thought it woul
Re: (Score:2)
"And yet a lot of them invent query languages/query languages similar to SQL. " - See, I think you're magically associating two classes of programmers. There are people, like myself that love the expressiveness of SQL over virtually any other language for data-set manipulation. Thus we would like as an optional to utilize SQL on even a
Re: (Score:2)
Simple key/value pairs work for some things. Most data cannot be managed reasonable as key/value pairs. And full text searches are entirely orthogonal. That involves searching through text rather than questions of semantic informa
Amazing (Score:2)
Cassandra has the goods for high available and optimized for non-financial data.
That said, I am amazed at how much time, money, and effort has gone into Twitter.
Now a distributed scalable super duper database will keep track of who is pooping. http://poop.obtoose.com/ [obtoose.com]
Re: (Score:1, Insightful)
Maybe in 5 years slashdot will get with it.
Do you realize how many years it took Slashdot to just remove their HTML table layout from Slashcode? I wouldn't bet on a major backend change for Slashdot, ever.
Re: (Score:2)
Re: (Score:3, Insightful)
Twitter's only moving to this new database written in Java because everyone else is.
Re: (Score:2)
Re: (Score:2)
Teradata seems to win typical OLTP and OLAP benchmarks. I would think for airline reservations and such that would be my choice of platform.
Re: (Score:2)
The problem is that 10K-12K transactions is 1/100 of what twitter need.