April 11th, 2005


History of Web Search from 1998 to 2005

Jan Pedersen at the Infonortics Search Engines Conference

Search has grown substantially, gone from a few million documents to at least five billion, at least 400 million queries per day worldwide, hundreds of thousands of machines engaged. The relevance, as defined by user search satisfaction, has become good (enough).

The very first web search engines were fairly simple word matching, the later generations more tuned for relevance. They used techniques like proximity weighting, anchor text (links pointing to the page), and incoming links (later including Google page ranks).

Note that the process of constructing a webwide search index is a very complex activity: lots of network traffic, analysis, reads and writes.

User Interface in the 1998 version of Infoseek are much the same as 2005 search results. The main changes are a "dynamic abstract" -- matching search terms in page context, and cached pages.

Other changes - targeted advertising successful business model, search can be a priority. Moore's law and a switch from big iron servers to relatively small servers in vast farms.