Stories
Slash Boxes
Comments

News for nerds, stuff that matters

Yahoo! Vs. Google: Algorithm Standoff

Posted by timothy on Wed Feb 25, 2004 07:53 AM
from the reverse-engineering dept.
An anonymous reader writes "There's a new report out from the guys who brought us the Google keyword density analysis. As they put it, "the goal of this analysis is to compare the keyword density elements of Yahoo's new algorithm with Google's algorithm." They compared 2000 low traffic, non-competitive keywords in the hopes of seeing the algorithms more clearly, without any possible search engine tweakings related to high-traffic keywords. Their findings are interesting. Should you go and rebuild your site based on these findings? Maybe not. It's worth a look though."
This discussion has been archived. No new comments can be posted.
Display Options Threshold:
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
  • by Anonymous Coward on Wednesday February 25 2004, @07:58AM (#8384856)
    Gee, aren't these the guys responsible for continually diluting the quality of search engine results? I'm getting really tired of sites that present one thing to search engines and something totally different to me.
    • by Anonymous Coward on Wednesday February 25 2004, @08:20AM (#8384988)
      As always, there are is a grayscale of good and bad search engine optimization. A good webauthor designs a site for the users, but keeps the workings of search engines in mind, too.

      Search engines need help with frames (if anyone can still find a good reason to use them). If you use Flash based navigation, you better make sure that you have a prominent document which links to all pages as well or search engines won't index them. It's also a good idea to use descriptive titles and put what's important at the top of the page. In other words, most good search engine optimization is exactly what you would do to make a site screen-reader or text-browser friendly.

      Then there's link-bombing, show-something-different-to-Google, white-on-white text, redirections, etc.
      It's quickly becoming so that you can't tell someone to optimize a site for inclusion in search indexes or they'll fall into the hands of this kind of scum. It's a little like the word "Hackers". Can't use that anymore without having to explain that you're not illegally breaking into other people's computers.
      [ Parent ]
    • by Anonymous Coward on Wednesday February 25 2004, @08:34AM (#8385068)

      I'm getting really tired of sites that present one thing to search engines and something totally different to me.

      Then complain about it. That practice is known as cloaking, and you can get sites blacklisted for it.

      [ Parent ]
    • by Araneas (175181) <(moc.sregor) (ta) (dnalilligp)> on Wednesday February 25 2004, @08:36AM (#8385076)
      It's an escalating battle. Someone hijacks a keyword that is highly relevant to your site so you have to figure how to overcome that and give users something that isn't porn or a crappy search portal.

      I think it's fair to say there are white hat SEOs as well as black hat hijack^H^H^H^H^H^H SEOs.

      [ Parent ]
    • That's what I wanted to submit to the Google programming contest, but it wasn't admittable:
      • Make a 2nd robot that retrieves a few full web pages (with graphics) per site claiming to be IE6 (or a normal Mozila), thus lying about it being from google.
      • Display the page in IE6 (or Mozilla), save the entire display as a bitmap image.
      • Run the bitmap image through an OCR program to extract the real text seen by the user
      • Compare this text with what the ordinary google robot sees.
      • If the text is completely different, lover the ranking
      This gets rid of all the blue on blue keywords, display:none keywords and others. I think it will come to that.
      [ Parent ]
    • SEO - SEM (Score:5, Informative)

      by peterdaly (123554) <petedaly@NOSPAm.ix.netcom.com> on Wednesday February 25 2004, @08:38AM (#8385091)
      (http://www.mythpvr.com/)
      As someone who does search engine optimization of his own sites, I believe there is an important distinction between ethical and non-ethical (spam) activities.

      Search Engine Optimization - doing all things possible to tell a search engine what your page is about while being balanced for humans to read as well. Ethical. Sometime considered spam when really the search engine returns poor results; usually due to the page you are looking for not being easy to understand for spiders.

      Search Engine Manipulation - trying to doing things to get search engines to return your page in results when the page may not otherwise be something the engine considers relevent or high quality. Showing something different for the search engine falls under this category, is commonly refered to as cloaking, and is against many search engines "rules" for designing pages. Not ethical, aka spam.

      -Pete
      [ Parent ]
      • Re:SEO - SEM (Score:5, Insightful)

        by silentbozo (542534) on Wednesday February 25 2004, @09:25AM (#8385429)
        (Last Journal: Sunday April 17 2005, @07:20PM)
        The problem is that telling the public what your site is about is equivalent to telling search engines what your page is about. Aside from meta-tags (which should really be all you need in order to communicate "additional" info to search engines), any change to your website to "optimize" for a specific type of search engine, and not for the general public, has the effect upgrading your page ranking AT THE EXPENSE OF NON-OPTIMIZED SITES.

        Here we go into the slippery slope that leads to situations like the tradgedy of the commons (where people tend to use up a resource because it isn't theirs), the hiring of lawyers (statistically, if one side hires a lawyer, they get better results, but if both sides hire lawyers they get the same settlement, only smaller because of lawyers fees), etc. It's the prisoner's dilemma - defect (ie, optimize) to improve my position, at the risk of everybody else defecting and earning worse returns than non defecting in the first place (ie, everybody stops using google because the rankings are screwed up and are no longer trustworthy.)

        Put simply, the moment any site tries to game the system, even just a little bit, they ruin the usefulness of Google. As it stands, I'm getting better results with Metacrawler now than with Google - something I wouldn't have said just a year ago. Don't even get me started on websites with javascript-redirect gateway pages, or the ones that scrape search-engine/newsgroup/eBay pages for text in order to boost hit counts, and then link back to similar pages in order to get higher link relevancy, OR the ones that take over abandoned domains in order to exploit the ranking generated by pre-existing links that point to the domain name...
        [ Parent ]
        • Re:SEO - SEM by RevDobbs (Score:3) Wednesday February 25 2004, @09:57AM
      • SE Dictatorship by Via_Patrino (Score:2) Wednesday February 25 2004, @01:10PM
    • Re:Search Engine Optimization Professional by Anonymous Coward (Score:1) Wednesday February 25 2004, @09:28AM
      • 1 reply beneath your current threshold.
    • Re:Search Engine Optimization Professional by scrytch (Score:2) Wednesday February 25 2004, @09:43AM
    • Re:Search Engine Optimization Professional by DavidStewartZink (Score:1) Wednesday February 25 2004, @10:25AM
    • Re:Search Engine Optimization Professional by Talinom (Score:2) Wednesday February 25 2004, @01:34PM
    • Bring it on by tomblackwell (Score:1) Wednesday February 25 2004, @10:19AM
    • That's right, mod me down. by Ayanami Rei (Score:3) Wednesday February 25 2004, @10:28AM
    • 2 replies beneath your current threshold.
  • Uhhh... (Score:1, Insightful)

    by Anonymous Coward on Wednesday February 25 2004, @07:58AM (#8384857)
    I don't get what the results of this test were. Did Google have better handling of the density, or did Yahoo? Is bigger better or does smaller win out?
    • Re:Uhhh... by ran-o-matic (Score:1) Wednesday February 25 2004, @01:42PM
  • If Yahoo wants my vote... (Score:4, Interesting)

    by PoprocksCk (756380) <poprocks@gmail.org> on Wednesday February 25 2004, @07:58AM (#8384861)
    (http://www.arklinux.org/ | Last Journal: Wednesday April 25 2007, @01:38PM)
    ...they'll have to get rid of all that junk on their home page. Much of the reason for my using Google is that its home page is simple, it loads quickly, and it is just so easy to _search_, which is what a search engine should be. Yahoo failed when it became a "portal" and tried to do too much by itself. If they could somehow reduce the size of Yahoo's page down to that of Google (that would mean getting rid of those ads, guys) then maybe I'd consider trying it.
  • And a User Friendly game to go along! (Score:5, Interesting)

    by Indras (515472) on Wednesday February 25 2004, @08:00AM (#8384869)
    Just grab a friend and a deck of cards, and you can play Yahoo vs. Google [userfriendly.org] at home.
  • I think (Score:5, Insightful)

    by Bishop, Martin (695163) on Wednesday February 25 2004, @08:01AM (#8384879)
    Google is way too embedded in everyones everyday life, it will just naturally be more widely used. When was the last time you heard someone say "Yahoo it"?
    • Re:I think by rosie_bhjp (Score:2) Wednesday February 25 2004, @10:25AM
  • Google Super Computer? (Score:4, Interesting)

    by YanceyAI (192279) * <yanceyai@yahoo.com> on Wednesday February 25 2004, @08:02AM (#8384888)
    Wasn't there a Slashdot article claiming that the Google servers may be the fastest super computer in the world, but they are so busy they couldn't run the benchmark? I can't find it now. If that's the case, how does Yahoo compete? By dividing the traffic? Can anyone link me?
    • Re:Google Super Computer? by PoprocksCk (Score:2) Wednesday February 25 2004, @08:11AM
      • Re:Google Super Computer? (Score:5, Informative)

        by /ASCII (86998) on Wednesday February 25 2004, @08:34AM (#8385069)
        (http://roo.no-ip.org/fish/)
        Your statement is not completely correct. There is nothing "fake" about a cluster based supercomputer. In fact, all sufficiently large supercomputers are cluster based. Many of them use special purpose, low latency NICS and switches, and proprietary communication protocols, but the underlying principle of a Beowulf cluster is the same as that of the Earth simulator.
        [ Parent ]
        • 1 reply beneath your current threshold.
    • 1 reply beneath your current threshold.
  • A layman's view (Score:4, Insightful)

    by EulerX07 (314098) on Wednesday February 25 2004, @08:03AM (#8384892)
    Yesyer I was hearing a colleague curse at his computer yesterday because he was looking for something specific.

    "Man, Goggle SUCKS now!, I'll try yahoo."

    "DAMN! Yahoo sucks even more!"

    I have to admit that I used to think google was incredible just after it came out, but nowadays I'm used to wading through 10-15 pages of results before finding something relevant to what I need.
    • Re:A layman's view (Score:5, Interesting)

      by Quaryon (93318) on Wednesday February 25 2004, @08:23AM (#8385003)
      Is anyone else getting so annoyed by pages which grab your keyword and then direct you to Amazon, no matter what the topic? Seems that every time I do a search on Google and find a site which looks interesting they're either just ripping Amazon's content or redirecting me there.

      Guys, if I wanted to go to Amazon I would just type "www.amazon.co.uk" into my browser.. If I'm searching on Google it's because I've either already looked at Amazon and didn't find what I want, or because Amazon is really not relevant..

      I've started adding "-amazon -kelkoo -dooyoo -pricewatch" and others to my Google searches recently which helps cut down the chaff a little, but doesn't seem to cut out all the Amazon ripoffs.

      Q.
      [ Parent ]
    • If you're "wading through 10-15 pages of results" by jbellis (Score:3) Wednesday February 25 2004, @08:23AM
    • Re:A layman's view by Anonymous Coward (Score:2) Wednesday February 25 2004, @08:41AM
    • Re:A layman's view by dubiousmike (Score:3) Wednesday February 25 2004, @09:00AM
    • Re:A layman's view (Score:5, Interesting)

      by pledibus (756395) on Wednesday February 25 2004, @09:13AM (#8385355)
      I think google's ranking system needs a major overhaul; various sleazy companies have become *much* too effective at fooling it. For example, below are the first three hits that I got by typing "prozac suicide" into google (I've deleted the URLs to protect the guilty :-). Most of the top 20 hits are similar to these.

      prozac suicide
      Prozac prozac suicide. prozac nation nude Viagra prozac hair loss Paxil
      prozac dogs Yasmin ssri prozac Propecia prozac ocd. ... prozac suicide. ...

      Prozac Suicide - Shopping and Discounts - PROZAC SUICIDE
      Prozac Suicide Prozac Suicide. Are you looking for Prozac Suicide? We've searched
      the internet for the best Prozac Suicide and we hope you enjoy what you find! ...

      Prozac Suicide
      Real Pharm - Lowest Prices & Fantastic Service - Prozac Suicide, ... Prozac
      Suicide Prozac Suicide. Prozac(R) is a selective serotonin ...
      [ Parent ]
    • Re:A layman's view by barthrh2 (Score:1) Wednesday February 25 2004, @01:43PM
      • 1 reply beneath your current threshold.
  • Pattern Recognition (Score:5, Interesting)

    by Space cowboy (13680) on Wednesday February 25 2004, @08:03AM (#8384893)
    (Last Journal: Friday April 27 2007, @02:20PM)
    This is essentially a problem in pattern recognition, and it's a damn hard problem to solve because of the disparity between the high-volume and low-volume words.

    Information is essentially the inverse of entropy. Entropy can be calculated, and you can use Bayes probability theory to get a hold on the information content of a given word within a set of words.

    What is difficult to do, and what search engines are trying to do, is measure the mutual information inherent between the set of pages that the word appears in, and the word itself, then apply that to all the words in the searched-for phrase; this is commonly called 'context'. This is plainly impossible to do for every given phrase, for every word combination, for every page indexed. The best you can do is use a statistical approach (and Bayes is your friend again) to come up with "good" matches.

    The problem with the statistical approach is the class unbiasing, since once you have wildly different statistical populations, your choice of context gets harder and harder - the "easy" standard models don't cope very well. You don't have the computational resources to do a good analysis, so you're essentially stuck between a rock and a hard place.

    This is why the google idea of strengthening the importance of a word depending on linked pages was such a good one - it "did" the hard work by relying on the entire planet to do it for them, by creating links. Of course, what one man can do, another can undo, and Google has got progressively worse over time. It's still by-far the best though, and my search engine of choice. When you look at the queries from search-sites, I get 100x as many from Google as Yahoo (next nearest)....

    People think searching is easy, and it is. What's really really hard is searching *well*.

    Simon
    • Re:Pattern Recognition (Score:5, Interesting)

      by Eivind (15695) <eivindorama@gmail.com> on Wednesday February 25 2004, @08:21AM (#8384989)
      (http://ekj.vestdata.no/)
      And what is even harder, as you sorta hint at, is searching well in a world where thousands of people do their damnedest best to game the system.

      Google doesn't only have to make sense of a great big mess.

      It has to make sense of a great big mess where a significant part of the pages are made *spesifically* to confuse Google, and where a part of those same pages gets tuned regularily in dedicated attempts at confusing whichever algorithm google use more.

      Most of the cases where Google returns poor results these days, it's obvious to a human observer that the bad results on top are *purposely* made to confuse Google. I've even seen pages that return one set of content if your user-agent is "Googlebot", and another, totally different content (dialer, etc) if your user-agent is anything else.

      [ Parent ]
    • Re:Pattern Recognition by fermion (Score:2) Wednesday February 25 2004, @08:45AM
    • Re:Pattern Recognition by psycho (Score:1) Thursday February 26 2004, @08:18PM
  • Keyword density?! (Score:5, Interesting)

    by Short Circuit (52384) <mikemol@gmail.com> on Wednesday February 25 2004, @08:04AM (#8384904)
    (http://shortcircuit.us/ | Last Journal: Sunday October 14, @02:01AM)
    When I search for something, I don't want to get a page that's a marketing front for what I'm trying to find, I want an informational, probably technical, page on the item I'm searching for.

    Such pages don't usually mindlessly repeat the keyword I'm searching for over and over again.
  • My little test.. (Score:4, Interesting)

    by CoolCat (594452) on Wednesday February 25 2004, @08:10AM (#8384934)
    Just typed in the company I work for name (8 employees). First hit on google, yahoo.. I gave up after 9 pages..
  • It's All Magic... (Score:5, Insightful)

    by photonX (743718) on Wednesday February 25 2004, @08:12AM (#8384945)
    I'm one of those greybeards who was writing college reports in the pre-BBS days, never mind the World Wide Web. Remembering back to when I used to spend a half-day of research in the library to mine info that now magically appears on my computer screen in ten seconds, well...it's hard to throw stones. I'm just happy the damned things work at all.

  • Teoma vs Google (Score:3, Funny)

    by Anonymous Coward on Wednesday February 25 2004, @08:14AM (#8384952)
    Search for "slash" in Google and the results are:

    1) Slashdot
    2) Slash's Snakepit ...

    Put the same "slash" keyword and search with Teoma [teoma.com]:

    1) Slash's Snakepit
    2) Slashdot ...

    Personally for this keyword search I feel Slash's Snakepit is more relevant and belongs at the top of the heap.
    • Vs Yahoo by paragon_au (Score:1) Wednesday February 25 2004, @09:09AM
    • Re:Teoma vs Google by Snowmit (Score:1) Wednesday February 25 2004, @09:12AM
    • Re:Teoma vs Google by Apiakun (Score:1) Wednesday February 25 2004, @09:43AM
    • 1 reply beneath your current threshold.
  • So that's what happened! (Score:5, Interesting)

    by peterdaly (123554) <petedaly@NOSPAm.ix.netcom.com> on Wednesday February 25 2004, @08:14AM (#8384953)
    (http://www.mythpvr.com/)
    I've been on vacation and away from internet and most mass media for a week. Got back on Monday and have noticed a drop in traffic to my web sites while I was gone. Didn't have a clue why. Well, now I know.

    I'll be watching this very closely. Inktomi (sp?) sucked, which is what this is based on. I think it's too early to tell right now if the results are any good. Along the same lines, it will probably take about 6 months for marketers to learn to effectivly spam the results, which is something Google has historically been very good at keeping at bay.

    This will be interesting to watch over the next few months.

    -Pete
    • Re:So that's what happened! (Score:4, Informative)

      by queen of everything (695105) on Wednesday February 25 2004, @08:31AM (#8385050)
      (http://www.seasonalwallpapers.com/)

      That's interesting. I've notice the reverse with mine. Slurp (Yahoo!'s bot) has been coming to my site almost hourly getting different pages for the past 2 weeks or so. I've also noticed a HUGE increase of referrers from search.yahoo.com. Usually all the referrers from search engines were from Google. Now, Yahoo! is much more frequent.

      Once yahoo changed over to Inktomi's search, I did several different searches for keywords or terms tha I want to be listed for. Surprisingly, I am ranked much higher on yahoo than Google right now for some things. I haven't changed anything in my code, its just interesting to see how the different search engines interpret the same thing.

      [ Parent ]
    • Re:So that's what happened! by Moeses (Score:2) Wednesday February 25 2004, @10:50AM
  • Warning: You are being watched! (Score:5, Interesting)

    by walter. (138617) on Wednesday February 25 2004, @08:14AM (#8384957)
    Looks like someone is counting the slashdot community. One of the links in this post points to
    http://www.searchguild.com/redir/o.php?out=http:// www.gorank.com/research/01072004_Google_Density_Re port.php
    So someone at searchguild.com is counting every slashdot visitor who clicks on that link! The unredirected link points here [gorank.com].
  • W3 compliance? (Score:4, Interesting)

    by valentyn (248783) on Wednesday February 25 2004, @08:18AM (#8384981)
    (http://valentijn.sessink.nl/)
    Slightly off topic: yesterday someone said that Google ranks W3-compliant pages higher than non-W3 compliant pages. I'm still confused. Could this be true?
  • Missing the google point? (Score:5, Interesting)

    by ItsIllak (95786) on Wednesday February 25 2004, @08:19AM (#8384984)
    (http://www.kallisti.co.uk)
    Isn't this missing the point of how google works? OK, so it measures the success, but it won't tell you anything (or much) about the actual search algorythm as google is actually basing the score not only on the page you link to but also pages that link to IT.

    Hence, it's an interesting read, and maybe you could draw your own preferences from what the weighting turns out to be in the listed cases, but it's not a very fair representation of how google works. *NB* I've no clue how Yahoo/Inktomi works, so I couldn't comment.
  • Sale sites. (Score:2, Insightful)

    by Bender Unit 22 (216955) on Wednesday February 25 2004, @08:21AM (#8384998)
    (Last Journal: Wednesday October 10, @06:37AM)
    I have seen that sites that does nothing but sells stuff, has gotten higher rankings lately. But maybe I just need to be more specific in my searches.
  • They are different (Score:4, Funny)

    by samsmithnz (702471) on Wednesday February 25 2004, @08:24AM (#8385016)
    (http://www.samsmith.co.nz/)
    For example if I search for me (Sam Smith), I show up 4th on Google, but 51st on Yahoo.

    I guess Yahoo really doesn't love me after all.
  • by G4from128k (686170) on Wednesday February 25 2004, @08:34AM (#8385066)
    While I know that various search engines use various core ideas in search, I would think that a better way to search would use multiple approaches. Some combination of link-based analysis, keyword analysis, expert analysis, cluster-analysis, etc. rather than a single "this-is-how-we-do-it-here" algorithm.

    The first big challenge in search is in disambiguating what the searcher really wants without requiring a long string of inputs. A multiple-algoithmic approach would let a search engine serve up hits gathered in multiple ways (e.g., hit number 1 was top ranked using mehtod 1, hit #2 was top ranked using methd 2, etc.). The search company could then see which algorithm provides the best hits for a given search (i.e., by watching which hits the searcher clicks on).

    The second big challenge is all the nasty spammers and SEOs (Search Engine Optimizers) who will try to use knowledge of any search algorithm to game the system and artificially raise their page rank for commerical purposes. This is probably one reason why Google cannot maintain dominance - any dominant search enegine attracts the concerted efforts of SEOs, thus ruining its search quality, thus ruining its dominance.

    Yet a multi-algorithmic search engine could create a moving target that frustrates SEOs. By rotating the algorithms and even using negative weights on some algorithm results, a multi-algorithmic search company could cause high-ranked pages to plummet in rank over time. One week, a heavily keyworded site (e.g., one listing every possible keyword in metadata) might be at the top of the list, the next week it is at the bottom of the list. This raises the cost to sites trying to game the system. (The search company might even reward or penalize sites that change structure to often to either find the freshest sites or penalize the efforts of SEO).

    There never can be one right way to do search.
  • They are search engine spammers (Score:3, Insightful)

    by Anonymous Coward on Wednesday February 25 2004, @08:44AM (#8385127)

    yeah trying to figure out how to get to the top of search engines by analysing keyword density so you can then construct copy text with fake entry pages or as the se.spammers call them "gateway" pages with 302 redirects via the useragent or constructing urls/with/the/keywords using ModRewrite

    we know what they are up to, spamming search engines peddling shite with their refferer links

    fuckers, these people are the reason 90% of search engines suck and who are rapidly poising google so in 5 years no-one can find shit without being taken for circlejerks and wading through shitty websites peddling porn,viagra and whatever shit is flavour of the month, if thats what the internet i see is gonna turn into then why the fuck do i bother

    and we link em here at slashdot
    i wouldnt give these people the time of day

    A>S
    • Insightful? by paragon_au (Score:1) Wednesday February 25 2004, @09:18AM
      • Re:Insightful? by 0x0d0a (Score:2) Wednesday February 25 2004, @10:21AM
      • 1 reply beneath your current threshold.
  • All I know... (Score:2, Funny)

    by Ironix (165274) <ironix@tr[ ]op.org ['oll' in gap]> on Wednesday February 25 2004, @08:51AM (#8385205)
    (http://www.trollop.org/)

    Is that I'm pissed off for suddenly loosing my ranking a month ago. I used to be in almost every spot for the top 30 results for the keyword "QQQ", but now I am below 100. =(
  • Cocks. (Score:5, Interesting)