How it all began
The World Wide Web was not always so, as we see from the screens of widescreen monitors. It developed gradually, but steadily, capturing the city and the country as a single information stream.
First, the Network became available to universities, libraries, and large companies. She was of a technical nature, comparable to mobile communications in the past decade, and often do not go beyond the limits of these institutions.
All that could be seen in the interface of "search engine" - a line that needs to key in the resource to which you want to move. Other options are not given, as is not given room for error - you have to know exactly what combination of numbers that takes the user to a simple page with text. However, the first resources not only provide an opportunity to review the information about the company, but also, for example, even order a pizza at home.
As soon as the network expanded, it took an objective necessity in making public the list of resources. In April 1994, for the purpose of advertisement and disseminating useful information, this catalog has been created on the website Yahoo. Web pages are divided into groups according to subject matter. With the expansion, Yahoo decided to do a new search, greatly increases the chances of a user. The algorithm allows to search for information in the descriptions of the resources that were present in the catalogs of Yahoo, excluding the sites are not added to his list.
To date directories of links are not interested, because they can not provide the user with that amount of resources that give the requests, for example, Googleili Yandex. If you compare one of the largest online directory DMOZ, the difference is the result turn out billions of links. For example, the catalog OpenDirectoryProject contains about 5,000,000 web pages, while Google has about $ 9 billion.
The first search engine, as we are used to seeing today was WebCrowler, which began operations in 1994. Since 1995, the leadership in the provision of search services take on the system Lycosi AltaVista. And in 1997, in the offices of Stanford University, outstanding programmers Larry Page and Sergey Brin created Google. This research project has become the most popular to date, the search engine in the world.
In the Russian analogue of the Google search engine is domestic programmers called Yandex. Her announcement was September 23, 1997 specifically for Russian-speaking inhabitants of the Network. In addition to Yandex in Russia are also systems Aport, Mail, KM, Rambler and, of course, the Russian-language version of Google.
Leaders in the number of requests in the scope of international interest in the search engines have become three competitors - Yahoo, MSNSearchi Google. Each company works with its own database and has a unique search algorithms. Many other search engines such as AOL, AllTheWeb, or Mail, are in varying degrees of bases and leaders do not hide it.
Highlights of the search engines
Let us to start the components that make up any search engine:
Spider or "spider" - a program similar to the browser dealing copying sites.
Crawler, «crawler" or "traveling" spider - a program that aims to automated reading all the links found on the Web page.
Indexer or "indexer" - a program dealing with the analysis of downloaded pages spiders.
Database, aka "database" - saves copied and processed page.
Search engine results engine, which means "the system of delivery of results" - displays the results found in the database.
Web server, or "web server" - he is in the process of user interaction with other services of the search engine.
The above features are inherent in search of each. The layout can be varied and distinguished one from another search engine, but in general, they remain as to Google, and in Yandex. For example, a combination of Spider + Crawler + Indexer, combined in one program by downloading specific resources, analyzes them and is looking for links to new resources.
Running the Spider («spider") - loads pages in the same way as any other modern browser. The only difference is that the "spider" is working with coding the page and does not display images and text, as does the browser. Instead of visual components, "spider" allows the user to view the dry html-code.
The tasks include determining Crawler route "spider" on the links on the site or on a predefined list of addresses. In other words, it makes all the relevant links on a web page. Thus, "crawler" is looking for new documents, even those that are not known to the search engine.
Technical work on the analysis page performs Indexer. The program keeps track of its elements and parts. Are analyzed text, style and structural features of a web page, its special features and service tags, written in html, etc.
Database («Database») ensures the safety of the results, the search engine downloaded for analysis. In some cases, database called the index of a search engine that does not affect the ability of the program.
Program Search Engine Results Engine is involved, as the name implies, the issuance of the results and rankings of web pages. Its algorithm generates the corresponding user requests the page in the order that it involves sorting. The algorithm allows you to display the most valuable information that would interest a user. With this component of the interaction optimizer that promote greater website ranking in the output. In the future, I will address the issue more factors relevant to rank results.
System Web server («Web server"), even if you do not know what it's called, is displayed on the main page of any search engine. It is derived by means of html-pages, and allows you to enter a search query. The same program is developing a display of results for a given query.