The Joys and Hype of Hadoop 55
theodp writes "Investors have poured over $2 billion into businesses built on Hadoop," writes the WSJ's Elizabeth Dwoskin, "including Hortonworks Inc., which went public last week, its rivals Cloudera Inc. and MapR Technologies, and a growing list of tiny startups. Yet companies that have tried to use Hadoop have met with frustration." Dwoskin adds that Hadoop vendors are responding with improvements and additions, but for now, "It can take a lot of work to combine data stored in legacy repositories with the data that's stored in Hadoop. And while Hadoop can be much faster than traditional databases for some purposes, it often isn't fast enough to respond to queries immediately or to work on incoming information in real time. Satisfying requirements for data security and governance also poses a challenge."
More accurate (Score:5, Informative)
Hadoop isn't a database.
It's a data processing system for massive quantities of data processed and distilled in large batches. If you're trying to treat it as a database, you're doing it wrong. The article is simple using Hadoop for the wrong purpose.
You use Hadoop to reduce large amounts of data into smaller more manageable collections of useful data, which can then be queried real time.
More accurate (Score:3, Interesting)
You're better off using k to process your data.
Source: we replaced hadoop with k. After a couple weeks of training, I was getting results faster than the high-priced hadoop contractors (most of them worked on the hadoop codebase, had written hadoop books, etc).
Re: (Score:2)
Hadoop isn't a database.
It's a data processing system for massive quantities of data processed and distilled in large batches. If you're trying to treat it as a database, you're doing it wrong. The article is simple using Hadoop for the wrong purpose.
You use Hadoop to reduce large amounts of data into smaller more manageable collections of useful data, which can then be queried real time.
They can't be queried in real time. Some set of data can be queried, relatively quickly. That's the whole point.
Hadoop takes your shit and fucks with it to the point where you can't ever be sure if you're getting truly live or complete data or not.
Re: (Score:3)
No, that's MapReduce. Hadoop is a distributed data store. Or a distributed file store optimized for large files.
It's not a database. It certainly isn't database management software. It won't manage your data for you. Even the myriad of tools built to run on top of it are nowhere near effective at it. Rather, they're more data consumers than data managers.
Well No Shi... (Score:5, Informative)
Hadoop, done right, takes many departments to work on and organize in a big enterprise. Small shops may have one guy who is both SA and Programmer who could get the job done enough to make a difference. Furthermore, you NEED a full install from a big vendor. Installing Hadoop from OpenSource is a nightmare, and the big vendor's make it painfully simple to get the job done quickly. Can you do it by hand? Sure. Do you have the time? Not when you have other projects to work on and you can spend the companies capital to get the install and config done in 1/10th the time.
Re: (Score:2)
This also assumes the data and the domain you're trying to apply Hadoop to are ones that Hadoop can effectively be useful for. A lot of PHBs and such are pretty ignorant when it comes to the problems that Hadoop can be applied to
Scale (Score:2)
"The setup, on an enterprise scale, takes thousands or tens of thousands of dollars in hardware"
You are off by at least two orders of magnitude, at last by any reasonable definition of "Enterprise".
An enterprise grade hadoop cluster that is dealing with enterprise workloads is going to start roughly in the mid-six figures and grow into the low 7 or 8 figures over time and scale. Scale is not cheap.
Re: (Score:2)
That's still an order of magnitude cheaper than stuff like Teradata.
Doesn't seem to be stopping the NSA (Score:4, Interesting)
Checkout the job postings in central Maryland near BWI: Java, Hadoop, TS/SCI with full scope poly. Hundreds of postings.
There is only one customer in near BWI that requires the last.
Apache Spark > Hadoop (Score:5, Informative)
For people who took a look a few years back, it has matured a lot from an interesting prototype to something I now use in production on my clients data. Documentation is still a bit sketchy for niche functions but it's improved a lot also.
https://spark.apache.org/ [apache.org]
Re:Apache Spark Hadoop (Score:1)
Running spark on hdfs seems to be a pretty good idea though, and you'll still need a YARN setup.
Or you can push your spark deployment on mesos.
It's simple... (Score:5, Funny)
The reason they're running into problems is they haven't fully embraced the synergy in B2B ROI cloud possibilities. If they utilize agile scrum development, they will be able to be on the bleeding edge of viral blog immersion while reaching convergence with real-time content management crowdsourcing.
Re: (Score:1)
You forgot vertical integration. :-p
Re: (Score:3)
The reason they're running into problems is they haven't fully embraced the synergy in B2B ROI cloud possibilities. If they utilize agile scrum development, they will be able to be on the bleeding edge of viral blog immersion while reaching convergence with real-time content management crowdsourcing.
The first ten words made sense and were in actual English. You're doing it wrong.
ETL (Score:2)
I remember Cloudera saying that most people use hadoop for ETL. Not sure if you've checked, but hadoop is like the ne plus ultra of ETL tools. It's worth a look if you have to transform lots and lots of data.
Re: (Score:1)
The problem is you still have to Extract data from other systems, Transform them to make them suitable for Hadoop and Load them in HDFS (or S3). Once that data is available to Hadoop, it becomes extremely powerful.
Practically all analytical systems have the
Re: (Score:2)
I remember Cloudera saying that most people use hadoop for ETL. Not sure if you've checked, but hadoop is like the ne plus ultra of ETL tools. It's worth a look if you have to transform lots and lots of data.
Um, for what purpose? After you use it as an "ETL" tool, the idea is that afterwards you can query it, analyze it, etc. Traditionally you used an ETL tool to get data into a database then used tools that spoke SQL to analyze the data. With Hadoop, you have to write all your ETL tools yourself. So using Hadoop as an ETL tool is really a bridge to nowhere.
Hadoop is webscale! (Score:1)
That means it is better.
Use a tool for what its good for (Score:2)
Hadoop is good at generally running massive queries of tons of data in a relatively efficient amount of time. I say efficient and not fast, becuase the requests can vary from well structured for grid data sets to massive bloated ugly queries that would be massive bloated and ugly in any DBMS environment. If you want to talk about regulation, etc.. I think you're batrking up the wrong tree with Hadoop. If you're concerned with regulation, seed the DB with unique though meaningless data when importing and avo
Re: (Score:1)
Re: (Score:2)
... back in my day we played nethack on the VAX-785!
I started out playing rogue on the Vaxen. Then there was plain hack. Those were the days. Still play nethack now and again.
Paywalled (Score:4, Informative)
Since when is it acceptable to post articles that are paywalled?
We're not even going to pretend to care about the article?
Re: (Score:2, Insightful)
Reading TFA before responding is considered bad form.
Re: (Score:2)
Yeah, obnoxious. People ought to browse submissions in private browsing mode or something. Then if they happen to have a sub to a paywall site they'd see the article the same way people who don't would.
Re: (Score:2)
You must be new here.
Sometimes, I don't even read the summary. Or most of the comments.
You must not be new here.
LOL (Score:1)
And while Hadoop can be much faster than traditional databases for some purposes,
If by "some purposes" you mean "idiots who don't know how to design a relational database", then sure.
Apple - $3B on crappy headphones. $19B on WhatsApp (Score:1)
Apple bought out Beats for $3B and change. They make middling, overpriced headphones that come in a variety of colors. Facebook dropped $19B on an app that sends messages. Facebook dropped $1B on a company that makes Polaroids on your smartphone.
$2B of investments into multiple companies that are working on a technology platform that provides methods for sifting though vast amounts of certain types of business data, running on low-cost, commodity hardware and backed by an open source project seems positivel
Free mirror on nasdaq.com (Score:5, Informative)
Embarrassingly Parallel (Score:1)
So, Hadoop is a framework for processing embarrassingly parallel (running the same function on a massive amount of aligned data chunked into pieces) tasks using Java.
This seems like a cluster-fuck (pun intended) to me that could get done as well or faster with an ordinary cluster environment with less software and memory overhead. For those in HPC, am I missing something? This also seems to have a very narrow scope of usage so you're getting a lot of mess for moderate returns.
Re: (Score:1)
Nope, you've missed nothing. Its over-hyped crap that only gained initial popularity because someone did it in Java, and enterprises like Java.
Re: (Score:2)
The main interest of Hadoop is that it makes it easy to do out of core computation if the computations are loosely coupled and are mostly IO-bound. For anything else, Hadoop is probably not the right tool and is overhyped and typically inefficient.