Open Source Solution Breaks World Sorting Records 139
allenw writes "In a recent blog post, Yahoo's grid computing team announced that Apache Hadoop was used to break the current world sorting records in the annual GraySort contest. It topped the 'Gray' and 'Minute' sorts in the general purpose (Daytona) category. They sorted 1TB in 62 seconds, and 1PB in 16.25 hours. Apache Hadoop is the only open source software to ever win the competition. It also won the Terasort competition last year."
100 bytes, 10 byte keys. (Score:5, Informative)
Re:When's it going to be 1.0? (Score:5, Informative)
The benefits of parallelizing everything! (Score:2, Informative)
2005 winner used only 80 cores and achieved it in 435 seconds. So with 800 cores what 2007 winner achieved is 297 seconds ?
Its not only number of cores its how the logic to use parallel nodes properly to do a particular task is important.
Hadoop won with 1820 cores (910 nodes w/ 2 cores each) at 209 seconds.
I'm all for better sorting algorithms, but eventually the cost of parallelizing something overtakes the profit made. That being said, Hadoop's internal filesystem made to be redundant, which is an important feature whenever you're dealing with large amounts of data.
Hadoop uses Google's MapReduce, by the way, whereas the competition didn't. It's nice to see MapReduce being used in a more public eye.
While better sorting algorithms -do- matter, I have to say that maintenance and running costs also matter.
I'd also like to see how a compatible C version of this software compares with the Java version. However, as I see it, the Java overhead seems fairly limited; sorting code is wonderfully repetitive, and I'd expect that it's already been optimized a fair amount.
By the way, the number of nodes and the hardware in the nodes for this Hadoop cluster is -optimized- for this contest.
Re:Overlords (Score:4, Informative)
I wouldn't be surprised if they came from Star Wars.
Actually, it came from Google. Sorta.
Apache Hadoop is an implementation of MapReduce that Google uses in their search engine. I believe the details were found in a paper Google released on it's implementation of MapReduce.
Not quite as impressive as it sounds (Score:4, Informative)
Google's sorting results from last yeat (link [blogspot.com]) are much faster; they did a petabyte in 362 minutes, or 2.8 TB/sec. They minute sort didn't exist last year, but Google did 1TB in 68 seconds last year, so I think it may be safe to assume that they could do 1 TB in under a minute this year. Google just hasn't submitted any of their runs to the competition.
From the sort benchmark page [sortbenchmark.org], the list the winning run as Yahoo's 100TB run, leaving out the 1PB run; that implies the 1PB run didn't conform to the rules, or was late, or something.
People have commented that this is a "who has the biggest cluster" competition; the sort benchmark also includes the 'penny' sort, which is how much can you sort for 1 penny of computer time (assuming your machine lasts 3 years), and 'Joule' sort, how much energy does it take you to sort a set amount of data. Not surprisingly, the big clusters appear to be neither cost efficient nor energy efficient.
Re:Overlords (Score:4, Informative)
"MapReduce: Simplified Data Processing on Large Clusters [google.com]." Jeffrey Dean and Sanjay Ghemawat, OSDI '04.
They wrote about it in Beautiful Code, too (great book). MapReduce isn't complex, in fact the name comes from a feature that a lot of functional languages provide (yeah, I know, it's not exactly the same thing).
There are many implementations of it. The wikipedia article is pretty informative: http://en.wikipedia.org/wiki/MapReduce [wikipedia.org]. I didn't know about "BashReduce"... Heh.
Re:Great! It's open source! (Score:4, Informative)
Here in the UK, the patent office has been issuing software patents for some time in "anticipation" of them becoming legal at some point in the future.
No, I don't understand that either.
Re:Overlords - Trivia (Score:5, Informative)
Comment removed (Score:3, Informative)
Comment removed (Score:3, Informative)