Facebook's Project Prism, Corona Could Ease Data Crunch 19
Nerval's Lobster writes "Facebook recently invited a handful of employers into its headquarters for a more in-depth look at how it handles its flood of data. Part of that involves the social network's upcoming 'Project Prism,' which will allow Facebook to maintain data in multiple data centers around the globe while allowing company engineers to maintain a holistic view of it, thanks to tools such as automatic replication. That added flexibility could help Facebook as it attempts to wrangle an ever-increasing amount of data. 'It allows us to physically separate this massive warehouse of data but still maintain a single logical view of all of it,' is how Wired quotes Jay Parikh, Facebook's vice president of engineering, as explaining the system to reports. 'We can move the warehouses around, depending on cost or performance or technology.' Facebook has another project, known as Corona, which makes its Apache Hadoop clusters less crash-prone while increasing the number of tasks that can be run on the infrastructure."
Bring You Own Shard (Score:5, Interesting)
I don't know why we don't abstract all this cloud data storage to a truly vendor neutral, competitive scheme.
Imagine if we as users could provide our own server (at home, or designate Amazon, Google, etc) for our cloud storage. Facebook becomes just one application and anything we post is encoded in JSON formatted files. Then we actually own what we post. Of course, Facebook can cache the data (since first post generally is the most needed) and only go back to your cloud for older data. But you'd have every wall post, every photo, all of your data shared. Then Facebook can give you 1gig free, but you can also buy 5gig from google and have them use that instead. It gives them infinite storage, they only need to manage their cache.
Re:Brilliant... (Score:5, Interesting)
This will continue to happen over and over until we decide to fix the problem and make better software.
We wouldn't need Facebook if computers were easier to use. There's so many simple tasks like sharing large files, streaming voice and video, identity verification, decentralized public & private backup among your friends and family, messaging and status, etc. which are completely doable using our own home computers -- Not relying on any 3rd party besides the ISP.
Data silos are retarding progress. The human race is decentralized, focusing data or traffic in one direction is the same bad idea as giving supreme power to only a few. The problem is that our culture should belong to us, not some centralized advertising engine.
With mobile computing on the rise, I see a repeated sentiment that you don't need a home computer anymore.... Well, I disagree. I think we need them more than ever -- They can stream us all the data we have and help smooth out spotty mobile connection issues -- Post your status to your own machine, and have your friends machines aggregate push it out to them. There really is no distinction between client and server at the packet level. We trade convenience and ownership for security only because the awesome machines we use still (mostly) have some really shitty software on them.
I hold conversations with my close friends and family online with only our ISP aware of the packets between us, with data that I know is strongly encrypted. I see and comment on the photos my brother takes hundreds of miles away, as soon as he's plugged in his camera (I'm in his highest trust group for that camera, but not his phone or GF's web cam). I've restored my grandmother's data from our decentralized online backup without paying a red cent to a 3rd party (other than our ISPs). We don't need an (untrustworthy) 3rd party server to negotiate connections, we share connection graphs between peers, and our routers COOPERATE with our PCs!
I've been working on this cross platform technology for years in my spare time, it's a shame there's no money in making computing BETTER... I'm only one man -- imagine what one of those multi billion dollar "technology" corporations COULD do, if doing so were profitable. Hell, imagine if I could work on this little project full-time... Or even just imagine what would be possible if anyone else actually gave a damn about their data?