Amazon Launches Public Data Sets To Spur Research 82
turnkeylinux writes "Amazon just launched its Public Data Sets service (home). The project encourages developers, researchers, universities, and businesses to upload large (non-confidential) data sets to Amazon — things like census data, genomes, etc. — and then let others integrate that data into their own AWS applications. AWS is hosting the public data sets at no charge for the community, and like all of AWS services, users pay only for the compute and storage they consume with their own applications. Data sets already available include various US Census databases, 3-D chemical structures provided by Indiana University, and an annotated form of the Human Genome from Ensembl."
Finally! (Score:4, Funny)
Re: (Score:1, Funny)
Score:0, Funny... why the fuck people are so humorless nowdays? To take the time to mod down a funny post... why don't you spare your mod points to mod down the 'frist psots!!1!!!' that became usual?
Btw, go ahead and mod me off-topic.
Re: (Score:2)
Re: (Score:2)
It did have a starting score of -1
Check off privacy (Score:1)
Re: (Score:3, Insightful)
One more step to a non private world CHECK
Depends on what you upload. Census data isn't private.
Re:Check off privacy (Score:5, Insightful)
One more step to a non private world CHECK
Privacy, as we have experienced in the last hundred years, is on its way out anyway. The sheer volume, immortality, and interconnection of, even publicly available, datasets inadvertently reveal information most of us would rather keep private. Much like how most people don't have a problem with beat cops regularly patrolling an area, but feel threatened by cameras monitoring, recording, analyzing, and storing information about the same public area.
That said, its here to stay. The data's here as long as we use credit cards for most purchases, use I-Pass(or similar) toll paying systems, carry GPS enabled cell phones, and expect the police to protect us from 100% of terrorist and criminal bogeymen. We might as well get some private research done, rather than leave it all to the government and big business.
Re: (Score:3, Insightful)
Privacy, as we have experienced in the last hundred years, is on its way out anyway.
It was only recently on its way in. For most of history people lived in small communities where everyone knew each others' business. Privacy only seemed to become a major concern when technology let us share information across large distances and with many more people.
I'm not commenting on whether that's a good or bad thing.
Not the same (Score:3, Insightful)
Which is very different from a large society in which some people know everybody else's business.
Even if this stuff is public, the time and money and knowledge necessary to use it will not be evenly distributed.
Re:Not the same (Score:4, Insightful)
Which is very different from a large society in which some people know everybody else's business.
Even if this stuff is public, the time and money and knowledge necessary to use it will not be evenly distributed.
Information has never been evenly distributed. In small communities it was the neighborhood gossip, the corner pharmacist, the village priest, or the county sheriff who knew everybody's business. The replacement of social capital with monetary capital is the only difference.
Those small communities had, however, a fast-acting, closely monitored feedback system. If someone abused their position of power and trust, it was caught quickly and it was easy to remove them from the loop. A similar system is needed now, only on a national, or worldwide scale. I think the only way to accomplish this, without going back to a pre-computer society, is to make sure that as much information about the watchers is as publicly accessible as possible. Hopefully, the same spirit that makes the OSS community so vibrant and quick to act will transfer to this new domain.
Re: (Score:2)
Well said. The feedback system, though, does exist in a few forms today. The press, for example, when they're doing their job, keep a watch on the government and report whatever they can find. The abusers of information can be removed from office with enough public interest. As another example, corporations can lose customers if they lose their trust.
Re: (Score:2)
Its true. Abuse by institutions already has a, relatively, effective feedback system. What I worry about is individual criminals or criminal organizations. There, the anonymity of access to the data prevents the feedback system from finding the person responsible for the abuse. I wouldn't mind having most of my personal data publicly available as long as I could see a unforgeable record of who had looked at it.
I guess I have the same attitude as the banks do regarding fraud. Its easy to commit, but really h
Re: (Score:3, Insightful)
Re: (Score:2)
There is a cloud-like structure developing out there where virtually EVERY electronic transaction will leave a permanent record. There will come a skynet-like point where we won't even have the option of simply restoring privacy.
I think we're well past the point of restoring privacy, as it existed in the middle part of this century, without tearing down the computational, networking, and legal infrastructure that we've created since the 1970's. As long as the capacity exists to record and search this data, and proves useful enough to those who collect it, we will live in a world where almost everything we do is monitored by machine.
The part that's hard to imagine is when these disconnected islands of, seemingly innocuous, data are
Re:Check off privacy (Score:5, Interesting)
The less privacy we have, the less we have to worry about our privacy. That sounds flip, and along the lines of "if you have nothing to hide..." but it isn't.
We want privacy primarly due to shame.
We have shame because we wear masks almost 100% of the time.
We wear masks don't want people to realize who we 'really are' either mentally or phyically.
We don't want people to really know us because we have been convinced to hold ourselves to standards that no one actually meets.
We hold ourselves to these standards because everyone else is wearing masks and while we can tell ourselves that 'they are just like us', it's hard to grasp that cognatively without actual proof.
If there were no privacy, no one could wear a mask. If no one were wearing a mask, we would realize that the standards we hold ourselves to are unrealistic. If we realize the standards we hold ourselves to are unrealisitic, we are freed from shame. If we are freed from shame, we no longer find privacy necessary.
Re: (Score:2)
Barack Obama called, he wants you as Attorney General.
Re:Check off privacy (Score:5, Insightful)
I certainly don't want to have to answer to the government anytime I say the word "bomb" or "terrorist" on the telephone, in email, or in an IM.
I also don't want some company complaining anytime they see me buy a product from one of their competitors.
I also don't want to have everyone on the internet knowing my social security number, address, license plate number, or telephone number.
That isn't because of "shame" that's because people can be assholes, and some people will abuse information. I don't care if people that I trust know these things, but I don't think shame or masks or whatever has anything to do with getting one's identity stolen, or having the government ensure you don't say anything bad about them.
That said, I don't think this public dataset business really affects individual privacy. This is more a database of already public, but hard to find, data, that doesn't contain personally identifiable anything in it.
Let's just hope they keep it that way.
Re: (Score:2)
But think for a moment why those items are annoyances.
You don't want to be heard saying certain keywords because the government has adopted the ineffective method of attempting to track terrorists by looking for these words. If they were forced to track EVERYONE, the government would quickly realize the folly of this course of action when it became apparent that not only did most terrorist already have ways around this, but the incidence of 'innocent' usage of these keywords was high enough that tracking th
Re: (Score:3, Insightful)
Likewise, the government wouldn't just change laws, instead they would (probably) just use the info
Re: (Score:2)
4chan'ers harass anonymously, do you think they would be still as willing to do so if you knew who they were? If EVERYONE knew who they were? How many people have you met who are complete dicks when they are under public scrutiny?
The "government" is not an entity, it's a group of people. People who themselves would be aware of the fact that they themselves are just as vunerable 'informationwise' as you are.
Information is only power when only one side has it. Once you realize that tagging someone with your '
Re: (Score:2)
Even with total transparency, everyone wouldn't know who they were, since no one could manage all the available information. People who cared would be able to find out, but if the harassers were part of a big enough group with enough resources that it was ineffective for their targets to retaliate against them, they could still get away with a lot. You'd have to do a l
Re: (Score:2)
To repurpose an Abe Lincoln quote:
"You can fool some of the people all of the time, and all of the people some of the time, but you can not fool all of the people all of the time."
Complete transparency only seems impossible because right now only a small group of people are watching everyone. It would be orders of magnitude harder to hide constantly something from everyone than it is to hide something from just a small group.
Re: (Score:2)
No, it seems impossible because it is.
We don't have a small group of people watching everyone now, we have no one watching everyone and everyone watching what they are interested in within their limits of access. Even if you assume everyone had access to everything (which is unlikely to be practical), you still won't have everyone watching everyone.
Re: (Score:1)
We don't want people to really know us because we have been convinced to hold ourselves to standards that no one actually meets.
You are confusing standards and ideals. There is a difference. Standards of behavior are met all the time. Sometimes you can actually find checklists of 'standard behavior'; things like etiquette guidelines and domain rules. It's also a sliding scale; your standard of behavior with your girlfriend is likely different than your standard of behavior with your boss.
Ideals of behavior are different; while people meet standards of behavior easily, no one really meets their ideals. This is a good thing and th
I don't know about all that. (Score:2)
Actually - I just want to seed my pirated movie collection, download midget porn, and read my boss's email and be the only person to know about it.
Re: (Score:1)
Is what you are smoking a legal substance?
Check off pants. (Score:2)
"We don't want people to really know us because we have been convinced to hold ourselves to standards that no one actually meets."
Yeah...well, I do have a bigger dick than everyone else.
Re: (Score:2)
I don't know why you have so much shame, but shame has nothing to do with my desire for privacy.
I like privacy to prevent abuse.
People knowing where my kids are, what my alarm codes are, what I research online, what medical information I have, and a million other things.
Society is a set of rules, this creates a common mask or expected behaviors. Ideally these behaviors are set by the same society that uses them. That way they can change as people do. Enforce societal rules, i.e. rules most people don't want
Re: (Score:2)
The first two items you listed, I would qualify as being 'protection', reasonable things to keep under wraps because other people out there have hidden lives you don't know about. And why don't you know about them? ^_^
The second two however, are what I would qualify as 'shame'. Granted you may feel I'm stretching the definition, but I can come up with two reasons why you wouldn't want people knowing about your medical information.
1. You have 'something' and don't want others to know because it might be emba
Re: (Score:2)
We want privacy primarly due to shame.
We hold ourselves to these standards because everyone else is wearing masks and while we can tell ourselves that 'they are just like us', it's hard to grasp that cognitively without actual proof.
If there were no privacy, no one could wear a mask. If no one were wearing a mask, we would realize that the standards we hold ourselves to are unrealistic. If we realize the standards we hold ourselves to are unrealisitic, we are freed from shame. If we are freed from shame, we no longer find privacy necessary.
Your three points there are dramatically off.
We want privacy because we want to be left alone, we want to only be known by those who know us, we wan to be a partner, not an object.
Other people hold us to these standards, and prosecute us both socially and (increasingly) legally.
If there were no privacy...there will always be privacy. Its just a matter of the equity of privacy. The government has a set up that hides themselves from scrutiny, for both legitimate and illegitimate reasons. They won'
Re: (Score:1)
Most of the data they've listed is available for public download elsewhere, such as the U.S. Census data. The 2000 Census is available from the Census bureau's website [census.gov].
Privacy? (Score:1)
Re:Privacy? (Score:5, Insightful)
This is true. But the easier it is to obtain datasets like these, the easier it is for anyone to do data mining and correlate the public (presumably non-identified) datasets with any private data they do happen to have.
Re: (Score:1)
But the easier it is to obtain datasets like these, the easier it is for anyone to do data mining and correlate the public (presumably non-identified) datasets with any private data they do happen to have.
Yes, but at least now we are all able to do data mining in large databases.
In what privacy is concerned, we're losing it, one way or the other. The only important question here is whether we are all going to lose the privacy (thus being somehow equal), or if there are going to be those which will keep most of their privacy (large corporations, politicians etc), while we (the mere consumer) is going to lose most of ours.
Large public databases like Amazon's at least tend to level the field a little, by p
Re: (Score:3, Insightful)
Yes, but at least now we are all able to do data mining in large databases.
This is absolutely the case.
The web has made vast amounts of information available, so you would think it would play into the "computers will bring about the age of big brother" that was so prominent during the 60s. But it hasn't. Instead, because everybody can afford computers and bandwidth, is had distributed power rather than concentrating it.
The rich and powerful already have access to vast datasets, and the computing and human power necessary to mine them. Things like Google and Wikipedia and blogs hav
Re:Privacy? (Score:5, Informative)
Re: (Score:2)
I believe the charge cost;which is fine for a government agency.
Re: (Score:2)
Is that still true? It was true in 1960, but it seems to me that their prices have risen and their costs have diminished.
Re: (Score:1)
That seems fine to me.
*badum tisk*
Selling EC2 service? (Score:4, Insightful)
Re: (Score:3, Insightful)
This just looks like a way to sell there cloud computing services. They provide the free data and you provide the monthly service fee.
I'd bet that's not quite how they think about it.
I once had the fortune to work on a small project for a guy who had built a pretty large software company and then sold it. He said that he always looked to do something interesting first, and then figured out how to make it not lose money, because money-losers aren't sustainable.
I don't know anybody at Amazon anymore, but from my pals who did work there, my guess is that AWS has a similar culture: they seek out the useful and interesting, and actually do the
Re: (Score:3, Interesting)
> If they had a culture that was mainly revenue-focused, I'd expect this idea to get shot
> down, because some penny-pincher would argue that they'd make more money from people
> uploading duplicates of these giant data sets over and over.
And a clever marketing man would counter that this is an opportunity to achieve lock-in by establishing exclusive access to a large number of datasets. Once people have built large, complex applications that use a number of these datasets in Amazon's environment an
Re: (Score:2)
And a clever marketing man would counter that this is an opportunity to achieve lock-in by establishing exclusive access to a large number of datasets.
You did note that these are public datasets, yes? And that you access them by mounting them like any other filesystem? Other than transferring a few hundred gig of data to somewhere else, there's no lock-in barriers.
Of course, that pursuit of openness could just mean that they have an extremely clever marketing person. But I've never met one quite that smart. Well, perhaps that's the wrong word. But all the marketing people I've met, even the ones I like a lot, remind me of the seagulls in Finding Nemo. Eve
Nope (Score:2)
This is just a benefit that they are giving to their customers.
They are storing huge, public, commonly-used datasets for their customers, free of charge. If you are a customer and want to use, say, census data, you don't have to waste your time uploading the data, and you don't have to pay the $0.10/GB to upload the 200GB of data. Amazon already hosts the data for you. You just run a simple command, and the data are now instantly available for you to use however you want.
If you are not an AWS customer, t
Standardization possible (Score:1)
Take a simple radix sort algorithm applied to a telephone directory. Radix sort works if the pre-allocation of slots matches the data. One example where it breaks down is if you used a Boston matrix [having larg
Catch 22 (Score:3, Insightful)
Note that on Amazon's website they say that you can only access the data if you're paying them to crunch numbers on their cloud computers.
That is, you can't just download the data off their sites, which would be the nice thing to do.
As such, this article is nothing more than a slashvertizement.
What's the license? (Score:3, Informative)
You'll recall that Amazon's "cloud computers" (ugh) are by the hour, and are pretty much root access to a VM. Unless there's a specific legal reason you can't, it's always possible to just download the data -- you'd just pay a bit for the time that instance must be up, and for the data transferred.
However, for those of us who already are using EC2, it's nice to not have to download the whole set -- which can be terabytes, for some of these -- and instead be able to simply mount it from wherever it is and wo
Re: (Score:1, Informative)
These data must be public to begin with, or they wouldn't host them:
If you have a public domain or non-proprietary data set that you think is useful and interesting to the AWS community [...] You must have the right to make the data freely available.
(How to share a public data set on AWS [amazon.com])
So I guess there isn't even a license. Free as in go grab 'em.
Re:Catch 22 (Score:4, Interesting)
Note that on Amazon's website they say that you can only access the data if you're paying them to crunch numbers on their cloud computers.
That is, you can't just download the data off their sites, which would be the nice thing to do.
And you know what you can do with a cloud computer, my little rocket scientist? You can set up a frickin' web server. And then you can download anything your precious heart desires.
Patent pending... (Score:5, Funny)
"1-Sick" -- Health Data
"1-Mick" -- Irish Census Data
"1-Dick" -- Porn Movies Database
"1-Lick" -- Lesbian Porn Movies Database
"1-Fick" -- German Porn Movie Database
"1-Hick" -- The George W. Bush Presidential Library catalog.
"1-Kick" -- Pharmaceutical Index
"1-Nick" -- Crime Data
"1-Prick" -- Copyright Law Legal database
"1-Trick" -- List of iKea-nu Reeves Movies.
"1-Tick" -- Camping Places Data set.
"1-Brick" -- The Lego Catalog.
"1-Thick" -- Obesity Index.
Re: (Score:2)
Oh, for mods points today. Well done (ethnic slurs aside).
Torrents? (Score:2)
Can someone make these datasets available to download as aggregate torrents, or are they available only once someone writes an application and gets it working on AWS?
With hard drives getting bigger and bigger, to me it makes sense to have lots and lots of local mirrors of this sort of data.
Re: (Score:2)
Can someone make these datasets available to download as aggregate torrents[...]?
That someone could be you! Steps to making this happen:
If you're serious about doing this and need advice on
Re: (Score:2)
Can someone make these datasets available to download as aggregate torrents, or are they available only once someone writes an application and gets it working on AWS?
Which part of the word "public" did you fail to understand? Of course you can download the datasets and do whatever you want with them. You can download them from Amazon, or you can download them from the original source.
AWS is just hosting them locally, free of charge as a courtesy to their customers. If you want to access/analyze census data as part of your application, the entire 200GB is available to you instantly with just a few command line calls. You don't have to upload or pay to store the data
Gnomes (Score:2)
Re: (Score:2)
Yes... yes you are.
You may want to seek help. :)
Re: (Score:1)
History repeats (Score:3, Funny)
>users pay only for the compute and storage they
>consume with their own applications
Everything old is new again!
Ah the good old days... when you had to PAY for cycles.... not like the young whippersnappers today with their "desktops" and "laptops" and more cycles than they know what to do with.
Sounds good (Score:1, Offtopic)
How many developers here have had to hunt around for a list of countries to populate a select box? Or chained select boxes for country -> county/state -> town? How nice would it be to have a central repository where you can download all (in any mixed selection of languages) in cvs/xml/etc?
Phillip.
Sounds like "Give us data so we can charge you" (Score:5, Insightful)
If the uploaded data is not available for download, but is only available to AWS applications running on Amazon's (paid for) compute service, then Amazon deserves nothing but contempt and an "Up yours" for this.
It seems that working for a living is out of fashion at Amazon. They expect people to supply them with resources so that they can charge them and others for their use. It's creative business bullshit, and not even remotely funny.
Amazon, how about you PAY BACK for the privilege of having the datasets uploaded to you by hosting them freely for the Internet community, and only on the back of that you charge for local, higher-speed access by AWS applications? Or would that be too "fair" for an Amazon business practice?
Re: (Score:3, Interesting)
Re: (Score:2)
I'm not trying to troll, and I assume I'm missing something, but why not put a microphone on and walk around collecting background noise?
If there isn't a technical reason this wouldn't work, you could keep track of what you are hearing and it would be much easier to categorize. Being a linear recording from a single source, you could tag 2:43-3:12 as "bus, traffic, swearing cab driver" without having to do it for 29 separate clips.
Re: (Score:2)
Re:Sounds like "Give us data so we can charge you" (Score:5, Informative)
If the uploaded data is not available for download, but is only available to AWS applications running on Amazon's (paid for) compute service, then Amazon deserves nothing but contempt and an "Up yours" for this.
Seriously? Or did somebody just put sand in your pancakes this morning?
As an AWS user, I think this is great. It means I don't have to waste time and money copying over a public dataset. When I read about this I fired up a virtual Linux box, attached the census data as /dev/sdb, and spent a couple hours rummaging. Total cost: $0.70. If I had had to copy everything over first, it would have been $20 in bandwidth, plus a long time waiting for the 200 GB to transfer.
You realize that these datasets are public, right? For the census one, you can already download it for free [census.gov]. Do you want Amazon to make it extra-super-free or something?
I presume it's the same for the others. But if not, you should put your money where your very active mouth is. It would take maybe 15 minutes work to get an Amazon server up and running, attach all the public datasets, and set up a web server.
I'm so very tired of people who say "somebody should do X!" but aren't willing to be that somebody.
Re: (Score:2)
A) Home Bandwidth is a sunk cost. Transferring it wouldn't ahve cost you more then a penny more then you are paying. Assuming you pay a flat rate.
B) Transferring the data would made it available to you for free, anytime.
Re: (Score:3, Insightful)
A) Home Bandwidth is a sunk cost. Transferring it wouldn't ahve cost you more then a penny more then you are paying. Assuming you pay a flat rate.
My time is not a sunk cost.
B) Transferring the data would made it available to you for free, anytime.
Most of these datasets are hundreds of GB in size. That's going to take a long time to download and it's going to mean buying a new hard disk and/or deleting your pornography collection.
The whole idea here is that if you are an AWS customer, and you're crunching a bunch of numbers, and need to crunch some census/genome/whatever data, you can type 'ec2-create-volume --snapshot <snapshotId>' and now that dataset can be attached to any EC2 instance. You don't have to wait to
Anyone asking about the data quality / validity? (Score:1)
Re:Anyone asking about the data quality / validity (Score:2)
I say: who vouches for the validity of the data set itself? I understand that some of the sets are already publicly available, but that doesn't mean all.
Did you even look at their public data sets page [amazon.com]? Every one of the public data sets they've listed has source information. They are already, at this very moment, providing information on who/where they came from.
The one I looked at also had extensive sets of README files explaining the source and format of the data, and I'd imagine it's true for the others as well.
"No charge"? (Score:2)
How is there "no charge to the community" when the data is accessible only to paying Amazon customers? I have no objection to them doing this, but the hype is a bit much. I guess the only "community" that matters to Amazon is the one consisting of Amazon customers.
AOL (Score:1, Funny)
I wonder if AOL will be submitting any "non-confidential" data...
All the genomic stuff is already free to download (Score:1, Informative)
plus much much more at:
at
http://genome.ucsc.edu/
http://www.ensembl.org/index.html
this is just a way to access it from amazon compute cloud.
Haw to analyze display survey data (Score:1)
I have been promised some simple survey data (bipolar survey) - what is the best way to analyze and display - I am into java/database/oracle and would like to use business intelligence techniques to test growing my career that way.
I realize there are lots of ways to do this, most of which would increase my skillset.
Best way to analyze these public data sets? (Score:1)
OK so you have the public data from amazon - how do you analyze it? (I realize the problem here is the amount of options available rather than being constrained down one path)