A Brief History of The Search Engine - Part 2 : Enter, the Search Engine
With the advent of spider-created catalog files, came the birth of several search engines. One of the first engines was introduced by the creator of the “WebCrawler” bot, also called “WebCrawler.” The search engine was later sold to America Online who later sold it to Lycos. As time passed, search engines evolved, making searches more complex and thus more efficient, delivering more and more accurate results.
Early search engines used very little data from website indexes to perform searches. For instance, the JumpStation engine collected only website header and title information, and used this to execute searches. As you might imagine, this information could be quite vague and result in misleading, incomplete, or all together incorrect query results. Even worse, the results that were returned were displayed in the order in which they were found by the engine. In other words, the results were in a random order, regardless of any other elements.
The problems with these primitive search engine techniques were clear, and the evolution of search methods continued. Excite was started in 1993, it separated itself from competitors by searching using statistical analysis of word relationships. Doing this produced exceptionally more relevant search results, leaving its competitors in the dust. Still, this was only the beginning. As companies had more and more bandwidth at their disposal, it opened doors for them to employ more complex search algorithms. For example, AltaVista was among the first search engines to allow natural language queries.
Although impressive at the time, these engines were only scratching the surface of search efficiency. In the months following the search engine boom, a monster would be introduced to the market. While it took much longer than its competitors to gain any attention, when it finally did, there was no turning back. When it did hit the scene, Google turned the world wide web upside down.
A Brief History of The Search Engine - Part 1 : A Problem and a Solution
During the mid to late 90’s, the world wide web’s popularity grew exponentially, as did its virtual wealth of information. However, this data was becoming lost in the vastness of its online environment. Developers were quick to recognize this problem, and find a way to organize the world’s data and make it accessible to users across the world.
The first step was to index the information on the world wide web. Doing this would mean going through every website in existence and identifying what information each site contained. What may have seemed like an unattainable goal to a single person, or even a large team was easily made a reality by a piece of software called a bot or spider. One of the first widely implemented bots was “WebCrawler.” Bots perform their task in a recursive manner, which is perfect given the organization of the web. A simplified demonstration of how a spider operates illustrates this recursive behavior: A bot is given a website as a starting point. The bot collects information contained on this site, and then searches for any link on the site. The bot will then follow the link found to another site. The bot then collects information from that site, searches for a link, follows it, and so on. Webcrawler did exactly that, just on a much larger scale. Over time, it built the first publicly-available full-text index of a large piece of the world wide web. In the following years, several more bots were created by entrepreneuring companies including Google.
Now that an index of the web’s information was publicly available, it was only a matter of time before developers would flood the net with applications capable of searching through this catalog, delivering results to user submitted queries. The world wide web was about to be introduced to the search engine, and it would not be long until this relationship would change the way the world found its information.