Search engines are the key to finding specific information on the vast expanse of the world wide web. Without the use of sophisticated search engines, it would be virtually impossible to locate anything on the web without knowing a specific URL, especially as the Internet grows exponentially everyday. Search engines for the general web do not really search the world wide web directly. Each one searches a database of the full text of web pages selected from the billions of web pages out there residing on servers. When you search the web using a search engine, you are always searching a somewhat stale copy of the real web page. When you click on links provided in a search engine's search results, you retrieve from the server the current version of the page.

There are basically three types of search engines: Those that are powered by crawlers or spiders; those that are powered by human submissions; and those that are a combination of the two.

Crawler-based engines send crawlers, or spiders, out into cyberspace. These crawlers visit a web site, read the information on the actual site, read the site's meta tags and also follow the links that the site connects to. The crawler returns all that information back to a central depository where the data is indexed. The crawler will periodically return to the sites to check for any information that has changed, and the frequency with which this happens is determined by the administrators of the search engine. Search engine databases are selected and built by these spiders. Although it is said they "crawl" the web in their hunt for pages to include, in truth they stay in one place. They find the pages for potential inclusion by following the links in the pages they already have in their database. They cannot think or type a URL or use judgment to "decide" to go look something up and see what's on the web about it. If a web page is never linked to in any other page, search engine spiders cannot find it. The only way a brand new page - one that no other page has ever linked to - can get into a search engine is for its URL to be sent by some human to the search engine companies as a request that the new page be included. All search engine companies offer ways to do this. After spiders find pages, they pass them on to another computer program for "indexing." This program identifies the text, links, and other content in the page and stores it in the search engine database's files so that the database can be searched by keyword and whatever more advanced approaches are offered, and the page will be found if your search matches its content.

Human-powered search engines rely on humans to submit information that is subsequently indexed and catalogued. Only information that is submitted is put into the index.

In both cases, when you query a search engine to locate information, you are actually searching through the index that the search engine has created; you are not actually searching the Web. These indices are giant databases of information that is collected and stored and subsequently searched. This explains why sometimes a search on a commercial search engine, such as Yahoo! or Google, will return results that are in fact dead links. Since the search results are based on the index, if the index hasn't been updated since a web page became invalid the search engine treats the page as still an active link even though it no longer is. It will remain that way until the index is updated.

So the same search on different search engines may produce different results. Because, not all the indices are going to be exactly the same; and it depends on what the spiders find or what the humans submitted. Moreover, not every search engine uses the same algorithm to search through the indices. The algorithm is what the search engines use to determine the relevance of the information in the index to what the user is searching for.

One of the elements that a search engine algorithm scans for is the frequency and location of keywords on a web page. Those with higher frequency are typically considered more relevant. But search engine technology is becoming sophisticated in its attempt to discourage what is known as keyword stuffing, or spamdexing. Another common element that algorithms analyze is the way that pages link to other pages in the web. By analyzing how pages link to each other, an engine can both determine what a page is about (if the keywords of the linked pages are similar to the keywords on the original page) and whether that page is considered "important" and deserving of a boost in ranking.

Recommended Search Engines: Table of Features

Search Engine	Google	Teoma	AlltheWeb Advanced	Alta Vista Advanced
Size, type (See tests and more charts.)	HUGE. Over 2 billion. Claims over 3 billion but about 1 billion are not fully indexed (i.e., cannot be full-text word searched).	LARGE. Claims to have 1 billion fully indexed, searchable pages, and 1 billion more partially indexed.	HUGE. Over 3 billion fully indexed, searchable pages.	LARGE, but smaller than Google or AllTheWeb.
Noteworthy features and limitations	Popularity ranking using PageRank. Limit of 10 words per search, excluding OR. Indexes the first 101KB of a Web page, and 120KB of PDF's.	Subject-Specific Popularity ranking. Suggests terms within results to refine. Suggests pages within results with many links.	URL Investigator to find out about a page. Conversion of weights and measures.	Full boolean searching and powerful searching within results using SORT BY box in Advanced Search. Basic search provides distracting commercial, paid, and directory entries.
Results Ranking	Based on page popularity measured in links to it from other pages: high rank if a lot of other pages link to it. Matching and ranking based on "cached" version of pages that may not be the most recent version.	Based on Subject-Specific Popularity, links to a page by related pages.	Also seems to use "importance" and links to pages. In Advanced Search , SHOULD INCLUDE gives higher priority to word or phrase in box. Each box read as a phrase. In Boolean Search, rank:word is supposed to rank by that term.	By the terms you specify in Sorted by box under Boolean search box. Relevancy ranked if left blank.
Language	Major Romanized and non-Romanized languages.	Major Romanized languages.	Major Romanized and non-Romanized languages. Allows you to specify matching character sets.	Extensive list includes major Romanized and non-Romanized languages.
Translation	In Translate this page link following some pages. To English from major European languages.	No translation.	No translation.	To and from English and other languages. Click on Translate following result.

In a meta-search engine, you submit keywords in its search box, and it transmits your search simultaneously to several individual search engines and their databases of web pages. Within a few seconds, you get back results from all the search engines queried. Meta-search engines do not own a database of web pages; they send your search terms to the databases maintained by search engine companies. The idea of meta-searching is much better than the reality in most cases. You would think you would save a lot of time by searching only in one place and sparing the need to use and learn several separate search engines. It depends a lot on what they search and how they organize the results. They cannot be better than the databases they query. There are two families of smarter meta-search engines: