Philipp Lenssen writes "Amazon's Alexa is releasing their search index (the same that powers the Wayback Machine) to developers via their new Alexa Web Search Platform. The Alexa framework is not for the weak of heart -- expect to learn how to use their C API, and expect to pay micro-amounts for requests and CPU cycles used -- but it also seems to be more powerful than the rival APIs from Yahoo and Google."
This discussion has been archived.
No new comments can be posted.
One dollar per CPU hour consumed. $1 per gig of storage used. $1 per 50 gigs of data processed. $1 per gig of data uploaded (if you are putting your new service up on their platform).
hi, i'll give you a friendly piece of advice. You are under no circumstances to read TFA before making at least one post. It's fine to read it, but you must make at least one wildass guess, and pretend to know what it's talking about.
Second, even if TFA did answer your question, you should again, under no circumstances be apologetic.
Alexa is notorious for spyware [google.com]. Use Ad-Aware to remove Alexa if you have Alexa installed. Programmers: I will boycott all Alexa-sponsored products and label them as spyware in turn if you use this "API."
It's a script that analyzes your web surfing. It IS spyware, except that you install it on purpose. In that regard, it's not spyware at all. It's lookware.
Disclaimer: I actually work at Amazon. I (have to) use a modified version of Alexa (and IE, ugh) every day for my job. Other than javascript conflicts that make some web pages slow, it works as advertised. In that it looks at and analyzes your browsing and reports it to Amazon, which in this case happens to be on the LAN.
Why is it spyware anymore than the Google Toolbar with Pagerank on, or for that matter the fact that there are google bugs all over the internet? Alexa is not bundled with anything and is very easy to unistall (use add remove programs).
Alexa - Tricks users into installing its software, or installs itself without permission - Actively tries to stop users from uninstalling it, forcing people to use a third-party app to remove it (Ad-Aware, etc.) - Tracks users
The first two make it scumware, the last makes it spyware. Google toolbar does track users, but warns them before doing so and only installs when users want it installed.
I'm, sorry,, but how does alexa "trick" people into installing it? Infact I was forced to close their privacy policy to install it, which made it quite clear that it would track my net usage. Since Alexa is uninstalled using "Add/Remove Programs" how is a third party app needed. The google toolbar with pagerank on tracks users in exactly the same way. (also upfront about it.)
Looks like I was wrong. I'm so used to seeing Alexa in AdAware logs that I lumped it in with all the other scumware out there. So far as I can tell, there have been two Alexa products: one that is bundled with IE as the 'Related Sites' feature (a Reg key that AdAware detects & removes, but which is reinstalled when the user repairs/reinstalles IE) and the toolbar. The toolbar seems well-behaved from all I've read.
Google, Yahoo, and MSN APIs are free but they do the searching for you and they have caps. Since you are paying Alexa, you can use it as much as you can pay. And, this new service does not offer search on it on its own. The Alexa Search Platform gives you only archive access [paraz.com] - you have to do indexing and search on your own.
"you have to do indexing and search on your own" Not strictly true, you have to do *ranking* on your own. Reading the docs it does let you reduce the document set, just not rank the finished result set. So you can filter the result set down to the matching documents, but which is the most important? Your algo decides.
Google and Yahoo give you finished results but only ones ranked by their own algorithms and then only the first 1000 result. Even then it's only 5000 query max for Yahoo and 1000 max for Google
Search is an important part of the Platform. Every document in the Data Store is indexed and included in the Platform's search engine, and this engine forms the backbone of all SearchBased collections.
Also, the crawl data includes pictures and movies, and the search engine metadata provides this useful field: CRITERIA,SEARCH FIELD = Adult content, Porn
The most obvious difference between the price of the Alexa beta and the Google Web API beta is, well, Google's is free as in beer, to a certain point (1000 queries per day).
Does anyone have any reasons that Alexa's API is better than, say, Google's?
I don't see any very good explanation of exactly how there system and aps working on it works. What I do read worries me. It looks like you need to guess upfront how much time your stuff will take against a partially opaque infrasturcture and scheduling model. If you overguess you get billed half the normal rate for the overage. no info I saw on what happens if you underguess. But it looks like your app is subject to being yanked. This reminds me of ancient batch processing days. I won't be writing s
That depends on your definition of what spyware is.
If you mean collecting data, then yes Alexa does it.
If you mean collecting personnal data, I don't think the toolbar does it.
Then what about Google? With AdSense running (almost) everywhere + your unique eternal Google ID, they surely collect a lot of data too. And with Google Analytics, they have also a lot of info.
So the question becomes: Is Google AdSense spyware?
I wouldn't call it spyware, as it doesn't feel to me like it's more spying than what a tourist does with a camera when visiting a country. It's all upfront, nothing hidden, much like Google's upfront privacy policy in clear text:
Google collects personal information when you register for a Google service or otherwise voluntarily provide such information. We may combine personal information collected from you with information from other Google services or third parties to provide a better user experience, including customizing content for you.
...
Google's servers automatically record information when you visit our website or use some of our products, including the URL, IP address, browser type and language, and the date and time of your request.
Nothing they try to hide deep down in some obscure EULA or anything. Sure, it's about collecting data, but there's a difference between collecting data, and collecting data by spying. The former is about doing it visibly, the other trying to hide it.
Besides, technically speaking, I'm not sure one should call a business model or an online service "spyware" anyway, as it's usually a term used for client-side software often piggybacking on another tool, that secretly phones home by using an internet connection.
Alexa provides the only global public ranking for websites, to me the data that alexa gives me for free unvaluable. Sure, I dont use their toolbar (firefox), and I dont find it very useful. They're giving something back. Thats not something you can say about the data google collects..
"We may combine personal information collected from you with information from other Google services or third parties to provide a better user experience, including customizing content for you."
To summarize: they can do what ever they want. But you're right: it is not hidden.
there's a difference between collecting data, and collecting data by spying. The former is about doing it visibly, the other trying to hide it.
Is displaying an ad something visible? You know they record every click (since they wi
If it is spyware, it's a useful one. Kinda like using a keylogger on your own machine to avoid losing that 10 Kb unsaved Word document. Search history is great - I can see what I was searching for a month ago and vaguely remember what I was doing that day, what I was thinking about etc.
For those who prefer "other" languages, they provide an app that (true to unix best practices) uses stdin/stdout for communicating with other programs:
The Data Retrieval API is written in C, so it may be natural for users to develop C applications against this API. However, the Platform features a utility named awsp_cat. This utility reads CIDs from stdin and writes the raw content to stdout. Users may develop applications in arbitrary programming languages to process the awsp_cat output.
Perl developers would be able to wrap this into their existing codebase in no time, assuming they want to pay the fees.
If I were Google, I would think about partnering with Amazon, or trying to buy them, I'm sure the idea would strike fear in the heart of many.
That certainly would give the Google Books project a different twist... It might
go away entirely, given that Google would be in essence competing with itself then,
giving users a reason not to buy books.
Now why would Google want Amazan. The company probably has one of the largest retained deficits in the world. It is and was amazing how many suckers Bozo got to invest in his company. Supposedly they are making a profit now; yeah 1c a share. I think Amazan is great to buy books from. After all they are selling most of their items below cost. Great for the customer. But as an investment? Forget it
Before arguing the price for a search, I would question the value of the data itself.
What's your opinion about Alexa ranks? Reliable?
IMHO, there is too few users of the Alexa toolbar. It is also quite biased (IE, Windows). So except maybe for the top 30,000 websites, I'm not sure about the reliability of the stats.
The diffences are major. Google's API gives access to search results or allows you to execute searches that can already be done through a browser. With G's API you can build apps like Gizoogle [gizoogle.com] and Google Rank Checker. Alexa's API goes beyond allowing users to execute search queries by giving up the content within the index. This is big news for anyone interested in building their own index or accessing content for other sites.
Someone can download billions of pages for several thousand dollars then use that to build their own search engine. Another user could be to mine the web for content such as email addresses(which would be bad). Alexa's announcement is a big shift and was bound to happen. Instead of getting crumbs from Yahoo & Google, they're giving up huge chunks of juicy data.
Ah, our old friends at Alexa... the ones who brought us the wonderfully flawed page ranking system that is based on data fed back from their IE plugin that records what pages you visit and builds rankings out of them. A quick review of their "top ranked sites" includes advertising providers like Doubleclick, and spyware providers like Claria. Depending upon the functionality of someone's IE browser is fatally flawed.
It seems some people (especially the author of the cited article) missed some very important points:
1. You have access to more than just the index - you have access to the crawled data, which is about 300 Terabyte. So, if you want to do something with the pages, you don't have to download them, you don't have to rely, that they are there - you can use the crawled data to do whatever you want.
2. The processing does not take place on your machine, but on the provided infrastructure. There is a Web-Interface, so you can administer your account, your jobs etc. You do not download any software from Alexa. You get an account on their Linux cluster and there you can compile and run your own arbritrary applications. You are able to provide these results in form of Amazon Web Services.
So, this is much more than Google, MSN or Yahoo offer, it's hard even to compare those services. Alexa is a complete different beast, and it's a huge beast.
The Internet Archive gets Alexa's old backup tapes of the web crawl and uses them to load up the Archive with page copies. The indexing systems are completely different. The Archive barely has an index.
Black Gold, Texas Tea.
Second time in an hour I've seen this term as it relates to a technology company.
The first was an optical networking [lightreading.com] company that is getting into Oil and Gas.
Pay? (Score:3, Funny)
Re:Pay? (Score:5, Informative)
Parent
Re:Pay? (Score:2, Offtopic)
Re:Pay? (Score:4, Funny)
i'll give you a friendly piece of advice.
You are under no circumstances to read TFA before making at least one post. It's fine to read it, but you must make at least one wildass guess, and pretend to know what it's talking about.
Second, even if TFA did answer your question, you should again, under no circumstances be apologetic.
Finally, welcome to /
Parent
Re:Pay? (Score:2)
How embarassing.
Re:Pay? (Score:1)
(it's like when you're good friend with a guy named Fleischer. it's OK to call him Fleisch)
Re:Pay? (Score:1)
Re:Pay? (Score:2)
Alexa? Nope. (Score:4, Informative)
Google's APIs [google.com] are better.
It's lookware. (Score:2)
What is the definition of spyware (Score:4, Insightful)
Parent
Re:What is the definition of spyware (Score:3, Informative)
- Tricks users into installing its software, or installs itself without permission
- Actively tries to stop users from uninstalling it, forcing people to use a third-party app to remove it (Ad-Aware, etc.)
- Tracks users
The first two make it scumware, the last makes it spyware. Google toolbar does track users, but warns them before doing so and only installs when users want it installed.
Re:What is the definition of spyware (Score:3, Insightful)
Re:What is the definition of spyware (Score:2)
Re:Alexa? Nope. (Score:1)
Oh, don't be so harsh, we are shuddering.
"Micro-amount" (Score:1)
Re:"Micro-amount" (Score:2)
Re:"Micro-amount" (Score:1)
There is something about that word that just bothers me... maybe it is all the porn sites out there who advertise their "micro" monthly cost.
Yeah, I hate those sites.
I mean, based on what I've heard.
Price (Score:4, Informative)
$1 per CPU hour ($.50 for unused hours)
$1 per GB/year
$1 per 50GB processed
$1 per GB downloaded
and $1 for every 4000 user requests.
This is just for search service, right?
And how do these prices relate to similar services?
Re:Price (Score:4, Informative)
Parent
Google & Yahoo API's are Results not Data (Score:2)
Not strictly true, you have to do *ranking* on your own. Reading the docs it does let you reduce the document set, just not rank the finished result set. So you can filter the result set down to the matching documents, but which is the most important? Your algo decides.
Google and Yahoo give you finished results but only ones ranked by their own algorithms and then only the first 1000 result. Even then it's only 5000 query max for Yahoo and 1000 max for Google
Re:Price (Score:2)
It does provide searching:
Alexa Web Platform User Guide > Search > Criteria > Overview [alexa.com]
Also, the crawl data includes pictures and movies, and the search engine metadata provides this useful field:
CRITERIA,SEARCH FIELD = Adult content, Porn
Coul
Re:Price (Score:2)
Re:Price (Score:2)
Re:Price (Score:2)
Re:Price (Score:1)
Does anyone have any reasons that Alexa's API is better than, say, Google's?
Re:Price (Score:2)
Spyware? (Score:2)
Re:Spyware? (Score:4, Interesting)
If you mean collecting data, then yes Alexa does it.
If you mean collecting personnal data, I don't think the toolbar does it.
Then what about Google? With AdSense running (almost) everywhere + your unique eternal Google ID, they surely collect a lot of data too. And with Google Analytics, they have also a lot of info.
So the question becomes: Is Google AdSense spyware?
Parent
Re:Spyware? (Score:5, Insightful)
Nothing they try to hide deep down in some obscure EULA or anything. Sure, it's about collecting data, but there's a difference between collecting data, and collecting data by spying. The former is about doing it visibly, the other trying to hide it.
Besides, technically speaking, I'm not sure one should call a business model or an online service "spyware" anyway, as it's usually a term used for client-side software often piggybacking on another tool, that secretly phones home by using an internet connection.
Parent
Re:Spyware? (Score:2)
Re:Spyware? (Score:2)
To summarize: they can do what ever they want. But you're right: it is not hidden.
there's a difference between collecting data, and collecting data by spying. The former is about doing it visibly, the other trying to hide it.
Is displaying an ad something visible? You know they record every click (since they wi
Re:Spyware? (Score:2)
God yes. However, slashdot loves google, so you will hear people explaining why spyware's actually a good thing in this case.
Re:Spyware? (Score:1)
Search history is great - I can see what I was searching for a month ago and vaguely remember what I was doing that day, what I was thinking about etc.
C not required (kinda) (Score:5, Informative)
The Data Retrieval API is written in C, so it may be natural for users to develop C applications against this API. However, the Platform features a utility named awsp_cat. This utility reads CIDs from stdin and writes the raw content to stdout. Users may develop applications in arbitrary programming languages to process the awsp_cat output.
Perl developers would be able to wrap this into their existing codebase in no time, assuming they want to pay the fees.
Amazon as a serious concurrent to Google? (Score:1, Funny)
Re:Amazon as a serious concurrent to Google? (Score:1)
That certainly would give the Google Books project a different twist... It might go away entirely, given that Google would be in essence competing with itself then, giving users a reason not to buy books.
Not that any of this would happen, of course.
Re:Amazon as a serious concurrent to Google? (Score:1)
Data Value (Score:4, Informative)
What's your opinion about Alexa ranks? Reliable? IMHO, there is too few users of the Alexa toolbar. It is also quite biased (IE, Windows). So except maybe for the top 30,000 websites, I'm not sure about the reliability of the stats.
Re:Data Value (Score:2)
Anyway the question remains: how good is the evaluation function of the search engine?
Re:Data Value (Score:2)
No, only the bias applies. For the top 30,000 websites, I think the daily sample is big enough to have at least a bit of meaning.
Do you really believe that MSN is more popular than Google?
Among the people that use the Alexa toolbar? yes. But of course, Alexa users are not representative of the internet population.
Shell access? Arbitrary C code? (Score:5, Interesting)
That seems a little dangerous, doesn't it?
Who is responsible for a security breach? (Score:5, Informative)
Man, I would hate to see who or what is held responsible.
Google API vs Alexa API (Score:5, Insightful)
Someone can download billions of pages for several thousand dollars then use that to build their own search engine. Another user could be to mine the web for content such as email addresses(which would be bad). Alexa's announcement is a big shift and was bound to happen. Instead of getting crumbs from Yahoo & Google, they're giving up huge chunks of juicy data.
Our favorite flawed rankings (Score:4, Insightful)
Re:Our favorite flawed rankings (Score:2)
Their rankings aren't flawed. They just don't represent what you want them to.
It's a raw count of GETs/POSTs, which includes pop-up advertising and such. It's not a ranking based on 'popularity'.
More than just an index (Score:5, Insightful)
It seems some people (especially the author of the cited article) missed some very important points:
1. You have access to more than just the index - you have access to the crawled data, which is about 300 Terabyte. So, if you want to do something with the pages, you don't have to download them, you don't have to rely, that they are there - you can use the crawled data to do whatever you want.
2. The processing does not take place on your machine, but on the provided infrastructure. There is a Web-Interface, so you can administer your account, your jobs etc. You do not download any software from Alexa. You get an account on their Linux cluster and there you can compile and run your own arbritrary applications. You are able to provide these results in form of Amazon Web Services.
So, this is much more than Google, MSN or Yahoo offer, it's hard even to compare those services. Alexa is a complete different beast, and it's a huge beast.Alexa's index does not drive the Internet Archive (Score:2)
Re:Black Gold, Texas Tea. (Score:2)