Slashdot is powered by your submissions, so send in your scoop

 


Forgot your password?
Close
typodupeerror
×
Programming The Internet IT Technology

How Hard Is It To Write Your Own Search Engine? 23

Posted by timothy from the can-barely-search-my-car dept.
kha0z writes "Anna Patterson, from Stanford University, overviews the difficulties that have to be overcome when attempting to develop and/or implement a search engine solution in this article in the ACM Queue Magazine. The article covers many issues dealing from data sources, to indexing, to ranking. How does Google make it look so easy?"
This discussion has been archived. No new comments can be posted.

How Hard Is It To Write Your Own Search Engine? More

How Hard Is It To Write Your Own Search Engine?

Comments Filter:
  • You start with a simple model, which they did years ago when the internet was a much much smaller and simpler thing, then work years and years to perfect your model, and voila, simple, no?

  • Search engine != entire web (Score:5, Insightful)

    by pauljlucas ( 529435 ) on Wednesday May 12, 2004 @10:28AM (#9126955) Homepage Journal
    Not all search engines are designed or intended for indexing and searching the entire web, and not everybody needs such a search engine. Often, people want to search their stuff: their documents on their local disks, their e-mail, etc.

    While writing a local search engine isn't trivial, it's a lot easier than writing a web search engine since all the scaling issues disappear -- I know: I wrote one [mac.com].

  • How does Google make it look so easy?

    Google has hundreds of millions of dollars. Google treats their 2000 (2500+?) employees pretty well, so those employees work hard and smart and put in 40-60 hour work weeks. Google started earlier then the other modern engines, and had some very good ideas.

    They focus on a small set of goals-- make it easy to search through a ton of information.

    Compare this to Yahoo and MSN, where search is really just one part of their business model (there is no Google Singles! or Google Games).
    • They focus on a small set of goals-- make it easy to search through a ton of information.

      Originally this was the case, but recently even Google has started forays into other services. Just click the "more>>" link above the search field on their front page.

      Froogle(shopping), groups(news groups), blogger, and of course there is the even popular /. subject, gmail.

      Unfortunately it seems they may be losing focus.
      • Well, most of the additional services still focus on searching and information. Information is also one of their core goals (I should have said this earlier). There still is nothing like Yahoo Travel or MSNBC.

        Froogle is still a search product, but with a focus on shopping.

        Groups is mostly still a search product (You can post also, so it's also about creating information). The service has been around for years (I think it's their second big project after web search). If I have a technical question, I often find the answers in Google groups. Blogger is new, but is similar to Groups in it's goals.

        Gmail is also largely about search. With search they can place ads in your email.

        Actually, I guess you can really say that Google is about using a good search technology to place highly targeted ads with the information.

        • Groups is mostly still a search product (You can post also, so it's also about creating information). The service has been around for years (I think it's their second big project after web search).

          Actually, Google Groups used to be DejaNews, and they bought the technology.

    • those employees work hard and smart and put in 40-60 hour work weeks

      This is an excellent opportunity to point out that 8 hours of those every week, mandatory, must be spent on a personal project not related to google's line of business.

      That one perk is the absolute best.

  • Picture here. (Score:3, Offtopic)

    by mosel-saar-ruwer ( 732341 ) on Wednesday May 12, 2004 @10:41AM (#9127106)

    I mean, be for real - who gives a damn about the article?

  • Andrew.S.Tanenbuam [cs.vu.nl]'s Book "Computer Networks" [phptr.com] deals this topic in a very elementary and a very good way. Any person,who is wondered by the Search Engine,should read that book. In the Chaper dealing with Application Layer,ast describes the basic data structures which constitute a web-search engine. You could better look at the presentation here [phptr.com].

  • Ask Tim Bray (Score:4, Informative)

    by grungeKid ( 4260 ) on Wednesday May 12, 2004 @11:25AM (#9127894) Homepage
    For a overview of the field that is full-text search (of which web search engines is an important, but not the only, part), you should read Tim Bray's essays on search [tbray.org]. He's been working on full-text search for a long time, knows his stuff and explains it in a very readable manner.
  • Thanks for the link to Google [google.com], submitter. I never would have found it otherwise!
  • I wrote a search engine once which was designed to index the entire web and some other basic Google-like features. It only took me a couple days (probably 20 total hours) to write the indexer and search engine. It searched quick and used very little disk space, but it was really slow to index due to CPU usage problems. If I had millions of dollars like Google, I could have bought a large number of machines to do the indexing, but I only had 3 machines. So after 2 months or so of attempting to index the web,

  • The crawl is hard, too (Score:5, Insightful)

    by blamanj ( 253811 ) on Wednesday May 12, 2004 @07:56PM (#9134660)
    ...harder than she implies.

    You have to deal with 404s, robots.txt, politeness (don't bring down someone's site by crawling too fast), redirects, content you can't handle (Flash, Javascript).

    The list goes on.
    • politeness (don't bring down someone's site by crawling too fast), redirects, content you can't handle (Flash, Javascript).

      I wrote my own crawler once. A just-for-fun-how-do-the-spammers-do-it kind of thing...

      I got around redirects and content by going straight via the network socket and looking at the response as pure text. Anything that fit the regex pattern for an email address got harvested.

      As far as politeness... I kept a circular growable queue (technically a linked list) of sites to visit. Eac

  • There is one open source search engine that seems to be up-and-coming. Nutch [nutch.org] is now powering Mozdex [mozdex.com], and it looks fairly impressive so far.

    Now, instead of the previous free-will donations [mozdex.com], you can support the project through purchasing very cheap sponsered listings [mozdex.com] that appear to the right of the results (similar to Google)

  • Anna Patterson was quoted in the Newsweek article on Google as a Google employee.
  • I had a go at writing one, and can summarize as follows. There are 4 essential parts...

    1) The crawler - goes out and retrieves pages
    2) The parser - parses the pages, finding links and text.
    3) The indexer - indexes the pages.
    4) The searcher - interprets the index in the context of a user's search request.

    None of these are especially hard to do simple versions of, but all of them are hard to do well.

Slashdot Top Deals

"Being against torture ought to be sort of a multipartisan thing." -- Karl Lehenbauer, as amended by Jeff Daiell, a Libertarian

Close