Prospects for the development of search engines. Internet search engines. How often do search engines change their algorithms?

KOVROV STATE TECHNOLOGICAL ACADEMY

Information and analytical information on informatics

on the topic: “Modern search engines, development trends of one of the Yandex market leaders”.

Completed by: 1st year student

3 academic groups

Makarov Ivan

Introduction. 3

Main part. four

Conclusion. eleven

Introduction.

Yandex is a Russian IT company that owns a search engine of the same name on the Web and an Internet portal. The Yandex search engine is the eighth largest search site in the world in terms of the number of processed search queries (1.290 billion, statistics for August 2009) and the second largest non-English search server after the Chinese Baidu.

The company's website was opened on September 23, 1997. 2000 is the year of the formation of Yandex. Yandex was founded by CompTek (the company that developed the Yandex search engine and supported it). The company reached self-sufficiency in 2002, turnover for 2006 - 72.6 million dollars, net profit - 29.9 million, for 2005 - 35.6 million dollars, net profit - 13.6 million.

The main and priority direction of the company is the development of a search engine, but over the years Yandex has become a multi-portal. In 2009, Yandex has more than 30 services. The most popular are: Yandex.News, Yandex.Fotki, Yandex.Toys and others.

The main office of the company is located in Moscow. The company has offices in St. Petersburg, Yekaterinburg, Odessa, Simferopol and Kyiv. In mid-June 2008, the company announced the opening of Yandex Labs - an office in the US, California.

Main part.

History of the company.

The Yandex.Ru search engine was officially announced on September 23, 1997 at the Softool exhibition. The main distinguishing features of Yandex.Ru at that time were checking the uniqueness of documents (excluding copies in different encodings), as well as the key properties of the Yandex search engine, namely: taking into account the morphology of the Russian language (including searching for the exact word form), searching taking into account distances (including within a paragraph, the exact phrase), and a carefully developed algorithm for assessing relevance (correspondence of the answer to the request), taking into account not only the number of query words found in the text, but also the "contrast" of the word (its relative frequency for this document) , spacing between words, and the position of the word in the document.

A little later, in the section "Tales" (observations on the content of the Russian Internet), the first tale of the Runet appeared - "Web - humanism or chernukha?". And in the "Numbers" section - the first estimate of the volume of the Runet, 5 thousand servers and 4 GB of texts.

Two months later, in November 1997, a natural language query was implemented. From now on, Yandex.Ru can be accessed simply “in Russian”, asking long queries, for example: “where to buy a computer”, “genetically modified products” or “international telephone connection and get accurate answers. The average length of a query in Yandex.Ru is now 2.7 words. In 1997, it was 1.2 words, when search engine users were accustomed to telegraphic style.

In 1998, Yandex.Ru introduced the ability to “find a similar document”, a list of found servers, search within a given date range, and sorting search results by last modified time. During this year, the "volume" of the Russian Internet has doubled, which led to the need to optimize search engines. Both then and now (with a volume of 200 GB), the search speed on Yandex.Ru is a fraction of a second.

During 1999, the Runet grew by an order of magnitude, both in the volume of texts and in the number of users. It was a year of rapid development for Yandex.Ru as well. The new search robot made it possible to optimize and speed up the bypass of Runet sites. Today, the Yandex.Ru search base is twice as large as that of its closest competitors.

The new robot made it possible to provide users with new features - search in different text areas (headings, links, annotations, addresses, captions for pictures), limiting the search to a group of sites, searching for links and images, and highlighting documents in Russian. There was a search in the categories of the catalog and for the first time in Runet the concept of "citation index" was introduced - the number of resources that refer to this one.

Throughout the year, work continued on the quantitative and qualitative analysis of the Runet. The NINI-index was opened (index "Inconsistency of Interests of the Population of the Internet"), showing the dynamics of changes in the interests of Internet users. A search Forum and a new service have been opened - a subscription to a request, that is, you can leave your request on Yandex.Ru and regularly receive information by e-mail about the appearance of new and / or modified documents corresponding to this request. By the beginning of the school year, the "Family Yandex" was opened, filtering search results from obscene language and pornography.

The origin of the word "Yandex".

Today "Yandex" is a word from the everyday life of an Internet user. It is often found on the Web “What, Yandex has already been canceled?”, “Loneliness is when Yandex is the first to congratulate you on your birthday”, “All questions to Yandex”. Many already think that this has always been the case. In a way, this is true - Yandex really appeared simultaneously with the mass Internet, when access to the network ceased to be the lot of selected technical specialists. But the very word "Yandex" is artificial, has its own authors and its own history.

In 1993, Arkady Volozh, the future CEO of the future Yandex company, and Ilya Segalovich, the company's future technology director, developed, as it turned out later, the main technology - the search for unstructured information, taking into account the Russian language.

The development had to be named somehow. Ilya remembers how he wrote down different derivatives of words describing the meaning of technology in a column. It quickly became clear that search (“search”) in Russian sounds too dissonant and you can’t make a successful combination based on it. The word index was more appropriate. So yandex appeared in the list of names - yet another indexer ("another indexer" or Language index). Both Ilya and Arkady liked the option - it is easy to pronounce, easy to write. In addition, Arkady suggested the letter "I" in the name - specifically Russian - Russian and leave it for clarity. So the word "Yandex" was invented. And the program file, respectively, was called yandex.exe.

In 1996, when search was offered to the general public for the first time as a technology, and not as part of a content product (before that, there were the International Classifier of Inventions and the Bible Computer Reference), the line of programs was called Yandex and this name was explained as Language iNDEX. The first programs in the line were Yandex.Site (search on one of your own sites - this product is now called Yandex.Server) and Yandex.Dict (morphological prefix for AltaVista, the only search engine that at that time knew how to somehow work with Cyrillic) .

But, of course, the word "Yandex" has become widespread since September 1997, after the launch of the search engine www.yandex.ru. Since then, users of the system have been offering us their interpretations. For example, Tyoma Lebedev, preparing to draw the first version home page Yandex website, said: “Ah, I understand, if the first “I” in the word index is translated into Russian, it will be “I”, that is, this will be “Yandex”. The authors honestly admitted that they did not think about it, but - a good interpretation, is accepted. Then someone on the Web suggested another option, seeing the two sides of the Internet, INdex and YANDEX. This word has already appeared derivatives, for example, Yandex employees are often called "Yandexoids" and less often - "Yandexians".

Search "Yandex".

Yandex search allows you to search the Runet, Uanet, and Kaznet (since October 14, 2009) for documents in Russian, Ukrainian, Belarusian, Romanian, English, German and French, taking into account the morphology of Russian and English and the proximity of words in a sentence. Since the beginning of 2006, Yandex search has been installed on the Mail.ru portal.

In addition to HTML web pages, Yandex indexes documents in PDF (Adobe Acrobat), Rich Text Format (RTF), Microsoft Word, Microsoft Excel, Microsoft PowerPoint, SWF (Macromedia Flash), RSS (blogs and forums) binary formats.

A distinctive feature of Yandex is the ability to fine-tune the search query. This is implemented using a flexible query language. So, for example, for the exclusion operation, you can specify the scope: the query A ~ ~ B will find documents (pages) in which A is present, but C is not present, and the query A ~ B will find documents where the word B is not present with the word A in one sentence. Similarly, the & operator looks for combinations keywords in a sentence, and && in the entire document.

Operator! allows you to disable morphology for a specific word as well!! allows you to specify the normal form, which allows you to get around some problems associated with homonymy. For example, the query !!Ivanov will find Ivanov and Ivanov, but not Ivanov.

By default, Yandex displays 10 links on each results page; in the search results settings, you can increase the page size to 20, 30, or 50 found documents. Sometimes the order of the sites on these pages may differ, since the databases for these results are not updated at the same time.

If there are a lot of links found for the query, the results page suggests limiting the search range - by region (that is, by IP range) or by date. If nothing is found for any word or words, it is proposed to replace it / them with similar ones (since the proposed options depend on the frequency of finding similar words, funny situations sometimes arise). Also, it is proposed to correct the words typed in the wrong keyboard layout.

From time to time, the Yandex algorithms responsible for the relevance of the issue change, which leads to changes in the results of search queries. The last officially announced changes were in March 2004, April 2005 and January 2007; according to unofficial information, there are much more of them (for example, the last one in August-September 2007).

In particular, these changes are directed against search spam, which leads to irrelevant results for some queries (less often for entire families of queries). Against search spam, which is not automatically filtered out, semi-automatic and manual moderation of the issuance (with the help of the so-called "white hat optimizers") is used, as well as a direct refusal to index "malicious" sites.

Owners, management and performance indicators.

More than 30% of the company, according to its own data, belongs to the investment funds ru-Net Holdings and Baring Vostok Capital Partners, 15% - to the Tiger Technologies fund, about 30% - to the founders of the company and 20% - to managers and other minority shareholders.

In mid-September 2009, it became known that the parent company of Yandex, the Dutch company Yandex N.V., issued a priority share, which was transferred to Sberbank for a symbolic 1 euro. The only right that a share gives is to veto the sale of more than 25% of the company's shares.

Management: Rkady Volozh - General Director, Ilya Segalovich - Technical Director, Elena Kolmanovskaya - Editor-in-Chief, Alexei Tretyakov - Commercial Director, Svetlana Kondrashova - Advertising Director.

All Yandex services.

Information retrieval:

Search and ya.ru

Directory - a directory of websites sorted by citation index. It is replenished manually by catalog editors, there is a possibility of paid registration.

News - The top news of the day, sourced from the mainstream media featured on the Internet. It is possible to search by news, as well as subscribe to news for a given search query.

Yandex.XML - using this service, you can make automatic search queries to Yandex in xml format.

Search on blogs and forums - search for resources that have an RSS-representation, as well as a rating of current queries, popular categories and news.

Market - search for offers for the sale of goods and services, selection of models.

"Meditative" search is the only search service in the world that has a "Search" button, but no search bar.

Dictionaries - encyclopedias, reference books, translation dictionaries.

Pictures - image search.

Video - video search.

Maps - maps of Europe and Russia, maps of major cities of the Russian Federation (up to the house), search on the map, as well as the ability to "wander" through the streets of some cities. [source?]

Addresses - search for contact information by the names of firms and organizations.

Poster - information about available events: cinema, theater, concerts, sports, clubs, etc.

Weather - weather forecast.

TV program - programs of central, regional and satellite channels TV.

Timetables - timetables for trains and planes.

Personalized:

Yandex.Video - video hosting and video search.

Mail - email.

Ya.ru is a blogging service.

Yandex.Fotki - photo hosting.

Spam defense - spam filtering.

People - free hosting for personal web pages, as well as a file storage service.

Yandex money - payment system, which allows you to pay for goods and services on the Internet.

Bookmarks is a bookmark storage system integrated with Yandex. bar."

Subscriptions - subscription to news.

Feed - online RSS reader

Yandex.Direct is a system for placing contextual advertising with pay per click.

The Cup is a regular Internet search competition.

Cities - Internet indices of Russian cities.

Tariff - search by tariffs of Internet providers.

Postcards

Spring - automatic generation of philosophical essays.

Internet - measures the speed of the Internet connection.

Mirror - A mirror of major Linux OS distributions, as well as FreeBSD and other projects.

Yandex. Local network - provides an opportunity to use all Yandex services not at the federal, but at the local rate.

Metrica - allows you to measure traffic, analyze user behavior and evaluate the effectiveness of advertising campaigns.

Software products:

Spam filter Spamodefense for corporate use (paid).

A program for searching Yandex Desktop Search files on a computer.

Ya.Online instant messaging program based on Jabber. It also allows you to receive notifications about new letters from Yandex. Mail, about new events from sites Odnoklassniki.ru and VKontakte.

The Punto Switcher program is an automatic layout switcher.

Widgets for operating rooms Mac systems OS X and Windows Vista, as well as for Opera browser: Search, Traffic, Clock, News.

Yandex ICQ - a special version of the ICQ client with symbols and integration of some services from Yandex.

Interesting facts.

1) The average length of a query in Yandex.Ru now is 2.7 words. In 1997, it was 1.2 words, when search engine users were accustomed to telegraphic style.

2) Yandex appeared before www.yandex.ru. The word Yandex was invented in 1993, and it was publicly uttered in 1996 and then meant not a company or a search engine, but a search technology on its own server and a morphological prefix to the Altavista.com search engine.

3) www.yandex.ru was launched to demonstrate the capabilities of Yandex technology, no one thought about making money on advertising.

4) The slogan “There is everything” was invented in 2000. In the same year, Yandex launched the first advertisement for the website on Russian television.

5) According to Yandex itself, about 80 percent of its audience is from Russia, about 3 percent from Europe, and just over 1 percent from the United States.

6) Some of the Yandex technical support staff operates under the collective pseudonym "Platon Shchukin".

Conclusion.

So, now we have complete information about Yandex. We know who manages it, how it works from the inside, what is the history of the company's development and much more. Now we can easily understand why Yandex is the leader in the Russian and global markets. I think the main reason for the success of Yandex is that the search engine copes well with the complexities of the Russian language. That is why search engines that were developed for English cannot index and rank Russian-language documents as well. The second advantage I see is the creative, friendly, cheerful slogans with which Yandex attracts users to use its services. Thematic pictures that Yandex places near its search line are much more accessible for a Russian user.

Leaders, trend growth in the number of proposals will continue. Those present today market electronic payment systems... more one milestone event: Paycash signed an agreement with the largest search engine system ...

Volga Federal District: contemporary status and prospects development(on the example of the Republic of Tatarstan)

Coursework >> Economics

... trends further development. ... leader. ... development one from the most important ... complex search and aerobatic... market. Development ... contemporary technologies, high-performance equipment, contemporary... supertoxicants; - development systems land monitoring...

Modern sociological problems of physical culture and sports

Abstract >> Sociology

To promote political leaders, parties, ... the total subject-object system socio-pedagogical ... creative search engine activity... market and the state. Market ... Trends development contemporary Olympic Movement Russia is one from ...

Trends development oil industry in the global economy

Abstract >> Economics

World market oil: trends development and... already carried out search-exploration work, ... Preliminary assessment. leader in world consumption... is one from essential elements contemporary world economic... world economic system, at the time...

To search in the index, the user must formulate a query and send it to the search engine. The request can be very simple, at least it should consist of one word. To build a more complex query, you need to use Boolean operators that allow you to refine and expand the search conditions.

The most commonly used Boolean operators are:

• AND - all expressions connected by the "AND" operator must be present on the searched pages or documents. Some search engines use the "+" operator instead of the AND word.
• OR - at least one of the expressions connected by the "OR" operator must be present on the searched pages or documents.
• NOT - the expression or expressions following the "NOT" operator should not (should not) appear on the searched pages or documents. Some search engines use the "-" operator instead of the word NOT.
• FOLLOWED BY - one of the expressions must immediately follow the other.
• NEAR - one of the expressions must be at a distance from the other, no more than the specified number of words.
• Quotes - Quoted words are treated as a phrase to be found in a document or file.

Prospects for the development of search engines

The search given by boolean operators is literal - the machine searches for words or phrases exactly as they are entered. This can cause problems when the entered words are ambiguous. For example, the English word "Bed" can mean a bed, a flower bed, a place where a fish spawns, and much more. If the user is only interested in one of these meanings, he does not need pages with a word that has other meanings. It is possible to build a literal search query aimed at cutting off unwanted values, but it would be nice if the search engine itself could provide appropriate assistance.

One of the variants of the search engine is a conceptual search. Part of this search involves using statistical analysis of pages containing the words or phrases entered by the user to find other pages that might be of interest to that user. It is clear that conceptual search needs to store more information about each page, and each search query will require more calculations. Many development teams are currently working on improving the performance and performance of these types of search engines. Other researchers have focused on a different area, which is called natural-language queries (natural-language queries).

The idea behind natural language queries is for the user to formulate the query in the same way as they would ask the person sitting next to it - without having to keep track of boolean operators or complex query structures. The most popular natural language search site today is AskJeeves.com, which analyzes the query to identify keywords that are then used to search the site index built by this search engine. This site only handles simple searches, but the developers are in a highly competitive environment developing a natural language search engine capable of handling very complex queries.

Modern search engines are the most powerful hardware and software systems, the purpose of which is to index documents on the Internet to provide data at the request of users.

To provide quality and relevant information, search engines have to constantly improve their ranking formulas. Ensuring maximum High Quality issuance for users and preventing the manipulation of it by optimizers - these are the key goals of the development of search engines.

At a time when search engines were just beginning to appear, their ranking algorithms were very primitive. Thanks to this, the most resourceful optimizers began to promote their sites so that they appear in the search results for the queries they are interested in. As a result, this led to the fact that resources that often did not carry the user any useful information, became the first, thereby relegating more useful sites to the background.

In response to these actions, search engines have become defensive, improving their ranking algorithms, introducing more and more variables into the formulas and taking into account all the new factors. Over time, this struggle between optimizers and search engines moved to a new level and contributed to the emergence of more advanced algorithms based, including on machine learning.

Stages of development of search engines:

As you can see from the diagram, the development of search engines and their algorithms goes in circles. Some create new algorithms, others adapt to them. It is difficult to say whether this process will ever stop, but personally I am inclined to believe that it will not. Despite the fact that search engine ranking algorithms have recently not only changed the significance of various factors, but also changed qualitatively, this does not frighten optimizers: their arsenal is constantly replenished with more and more new techniques.

How often do search engines change their algorithms?

Let's turn to the main search engine of the Runet - Yandex. Qualitative and fundamental changes in the ranking formulas in it occur on average once a year. Not so long ago, Yandex introduced a new search platform called Kaliningrad. Its essence lies in the formation of personal results for each user based on his search history and preferences.

In addition, do not forget that every search engine, including Yandex, constantly has “twisting” ranking formulas, when in automatic or semi-automatic mode the influence of certain factors is underestimated, while others, on the contrary, are increased. All this is done with only one goal - to maximize the search results, ridding it of sites that do not meet the needs of users, and thereby increase its relevance.

Looking at the changes in the Google search engine, you can see that the transformation of the ranking formula also occurs constantly, and Google itself reports hundreds of small changes from year to year. But if we are not talking about the ranking formula, but about the filters that help Google clear the search results from low-quality sites, then new versions of algorithms, such as Panda or Penguin, appear every 3-6 months.

The answer to the above question can be as follows: search engines are constantly improving their ranking algorithms, and cardinal changes occur on average once every 6-12 months.

Which search engine algorithms pose a real threat to promotion?

I would like to answer “at the rally” - none, but still, let's figure it out. And for this we need to ask ourselves the question - do search engines make it their goal to prevent search promotion?

I think not. There are several reasons for this:

1. Optimizers help search engines improve their algorithms, which ultimately leads to better search results. After all, if there were no optimizers, then search engines, most likely, would have stopped in their development in the year 2000.

2. Without optimizers, the results for many commercial queries would look like a collection of abstracts and useless informational articles.

If search promotion did not exist in principle, then it would not make sense for search engines to grow and develop as intensively as they do now.

Thus, we come to the following conclusion:

Search engines and SEO are closely and inextricably linked with each other. That is why, following the rules established by them, you can absolutely not be afraid of algorithms, because PS do not aim to destroy SEO as such.

Development of search engine services

Speaking of search engines, do not forget that Yandex, Google or Bing have their own services designed to help users. In addition to search results, over the years of evolution, PSs have studied the behavior of their users in order to increase satisfaction with the results of issuance.

Actually, for this, the Yandex search engine came up with a mechanism for the so-called. "Sorcerers" who help the user quickly get an answer to their question. So, for example, when you enter the query "weather forecast", Yandex will display weather information on the search results page directly on the search results page. current date, thus saving the user from the need to navigate through the results of the issuance.

Other search engines, such as Google, went further and instead of "Sorcerers" offered a more interesting solution - the "Knowledge Graph".

“Knowledge Graph”(from English Knowledge Graph) is the first step on Google's path to intelligent search. Thanks to this innovation, the search engine displays in the search results not only standard links, but also direct answers to user questions, a brief reference about the query object and information about facts related to it. Technically, the “Knowledge Graph” is a semantic network that links together various entities: individuals, events, spheres of life, things, categories. The information base for the “knowledge graph” is a number of sources: the open semantic database Freebase, Wikipedia, the CIA open data collection and other sources.

What conclusions can be drawn, you ask?

The answer is simple: search and search services will continue to develop towards quick and relevant answers to user questions, providing the opportunity to get all the necessary information directly in the SERP (issue) and eliminating the need to go to other sites.

There is an opinion that search engines, by their desire to answer the user's question here and now, can destroy search engine optimization, becoming a kind of global knowledge base. But such fears are groundless, because in order to become global knowledge bases, they need information, and it is stored by the very sites that the very optimizers work on, who are involved in the fact that search engines do not stand still, but constantly evolve.

As you can see, both SEO and search engines are links in the same chain that cannot exist without each other. Therefore, thoughts about the imminent death of SEO are unfounded. It is possible that search engine optimization evolves over time, for example, in consulting, but certainly will not die. I wish you all a successful promotion to the TOP!

Introduction

3.1 Gopher

3.2 WAIS

3.4 AltaVista

3.5 Opentext

3.6 Infoseek

4. Search robots

5.1 Rambler

5.2 Yandex

5.3 Port

6.1 Google

6.2 Yahoo

7.1 Baidu search engine

8. Prospects for the development of search engines

Conclusion

Bibliography

Introduction

Each user on the Internet can find a lot of diverse and interesting information, as well as to use all the richest possibilities of the network. The selected topic of the essay is very relevant today, because. Search engines are indispensable today, due to the extremely frequent visits to the World Wide Web. Internet resources have become a tool for the daily work of people of many professions. The rapid growth of information on the web has made it an ocean of diverse data, the importance of which grows in proportion to their volume. According to experts, the volume of information transmitted via the Internet is doubling every six months. Every day, millions of new documents appear on the web, and it is natural that without search engines, the vast majority of them would remain unclaimed, they would not be found by anyone at all, and all that huge amount of information would turn out to be useless. There was a need to create such tools that would make it easy to navigate the information resources of global networks, quickly and reliably find the information you need. There are special search tools on the Internet. A few years ago, there was such an opinion: the Internet has everything, but it is impossible to find anything there. However, with the advent and rapid development of search directories, search engines, and all kinds of search programs, the situation has changed, and now urgently needed information can sometimes be found on the Web faster than in a book lying on the table.

Unfortunately, search engines often fail to accurately and fairly interpret resources. As a result, sites "far" from the issue being solved often appear in the first positions of the search. At the same time, resources that are of real use are "overboard" in the search.

search engine internet robot

The reason for this situation is simple and lies in the technology of obtaining and presenting results by search engines. Paradoxical as it may seem, this is not the fault of search engines, since they are obliged to hide the rules for building search indexes. It is the fault of the technology itself in organizing the search

A search engine is software that provides access to a collection of semi-structured information. Orientation to semi-structured data, i.e. data that cannot be represented as a relational table distinguishes a search engine from a DBMS.

In this definition of a search engine, information of various kinds is implied, i.e. text, audio, video, images, etc. However, it should be noted that it is textual data that is ideal for describing the full functionality of a search engine, because multimedia information search algorithms are primarily based on text search algorithms.

The main task of the search engine is to minimize the time spent by the user searching for the desired information. The question is, what information does the user want? In some circumstances, relevant information can be defined as all information in the database that is relevant to a query. Traditionally, two main characteristics are applied to the search engine: accuracy and completeness, or rather, their dependence. Each time the user submits a query to the system, thereby initiating a search, all documents in the search engine's collection are divided into four parts. Accuracy determines one aspect of search, namely how well a search engine is able to minimize the time a user spends searching for relevant information. this request information. While completeness determines another aspect - how well the system is able to find information relevant to a given query. One can choose the optimal query(s) when every document found is relevant and every relevant document is found.

Search engines play a very important role when using the Internet. There is so much information on the Internet that its search is already turning into a separate task and takes a lot of time. Search engines give thousands of links per query instead of a few pages where there really is necessary information. Users of the World Wide Web, realizing the benefits provided by the ability to analyze spatial data, need a tool that allows them to quickly and easily search and access digital images of the terrain and other spatial information concentrated in many government, commercial and academic organizations.

1. The history of the development of search engines

One of the first ways to organize access to information resources network was the creation of directories of sites in which links to resources were grouped according to the subject. The first such project was the Yahoo website, which opened in April 1994. After the number of sites in the Yahoo directory increased significantly, the ability to search for information in the directory was added. This, of course, was not a search engine in the full sense, since the scope of the search was limited only to the resources present in the catalog, and not to all resources on the Internet.

Link directories were widely used in the past, but have practically lost their popularity at the present time. The reason for this is very simple - even today's catalogs, containing a huge amount of resources, provide information about only a very small part of the Internet. The network's largest directory DMOZ (or Open Directory Project) contains information on 5 million resources, while the Google search engine database consists of more than 8 billion documents.

The first full-fledged search engine was the WebCrawler project, which appeared in 1994.

In 1995, the search engines Lycos and AltaVista appeared. Last long years was a leader in the field of information search on the Internet.

In 1997, Sergey Brin and Larry Page created Google, the most popular search engine in the world today.

September 1997, the Yandex search engine was officially announced, the most popular in the Russian-speaking part of the Internet.

Currently, there are 3 major international search engines - Google, Yahoo and MSN Search, which have their own databases and search algorithms. Most other search engines (of which there are a lot) use the results of the 3 listed in one form or another. For example, AOL search (search. aol.com) and Mail.ru use the Google base, while AltaVista, Lycos and AllTheWeb use the Yahoo base.

In Russia, the main search engine is Yandex, followed by Rambler, Google.ru, Aport, Mail.ru and KM.ru

AltaVista-search system. The name "AltaVista" literally translates as "view from above".

Initially, the AltaVista search engine was a true innovator in the creation of search technologies. In 1995, Alta Vista was created as one of the equipment elements of the Digital Equipment Corporation (DEC) research laboratory. Having appeared, the AltaVista search engine quickly gained recognition from users and became a leader among its own kind. The main merit of the AltaVista system is the support for many languages, including Chinese, Japanese and Korean. Indeed, in 1997, not a single search engine on the Web worked with several languages, especially with rare ones.

In 1998 Compaq Computer Corporation bought DEC (along with AltaVista). And already in early 1999, AltaVista received the status of an independent division. That same year, Microsoft licensed the AltaVista search engine for use on its MSN site. Many people immediately began to use the services of indexing large amounts of information and the ability to instantly search in huge databases. At the same time, the address of the search engine remained the same - altavista. digital.com.

And typing altavista.com in the address bar led to the site of AltaVista Technology. As a result, the popularity of the search engine resulted in a huge influx of visitors to the AltaVista Technology site and a loss of potential search engine users. As a result, the altavista.com domain was purchased by Compaq for $3.35 million in August 1998 (the largest deal of its kind at that time). Despite this, Compaq never managed to profit from the search engine. Therefore, in June 1999, negotiations began between Compaq and CMGI Corporation to form a strategic network alliance, in which AltaVista was sold to CMGI. On August 19, 1999, CMGI Corporation announced that it had acquired an 83% stake in AltaVista from Compaq.

In February 2003, AltaVista was acquired by Overture Services, Inc., which was acquired by Yahoo in July 2003. Since May 2011 AltaVista has switched to Yahoo search technology.

The AltaVista search engine, on the other hand, sought to be a one-stop portal that included an online store, radio station, forums, chat rooms, personal photo albums, and more. But, due to huge cash injections, due to competition with other giant portals and published criticism from the same competitors, 2001 passes for the company under the motto of abandoning claims to portal status and "returning to the roots .

The company turned its activities in a different direction. Now www.altavista.com promotes its search engine to individual Internet users and licenses search technology to businesses, including for use on intranets. The main source of funding for the consumer version of the AltaVista search engine was advertising revenue, including from the most popular ones. For example, real search results are now placed after the link, for which AltaVista is paid by the owner of the respective resource.

Along with its efforts to become a portal, AltaVista continued to improve its search technology.

Also, another source of income for AltaVista is the development of corporate internal search engines.

Despite the obvious lagging behind the competitors, www.altavista.com is absolutely confident in its abilities. We hope that the Alta Vista company will fulfill all the plans and successfully return to its roots. . The AltaVista search engine (www.altavista.com) won the hearts of all Internet users at an early stage of its existence. Its history is a classic example of combining good technology with vague positioning.

2. How search engines work

Search and structuring tools, sometimes referred to as search engines, are used to help people find the information they need. Search tools such as agents, spiders, crawlers and robots are used to collect information about documents located on the Internet. These are special programs that search for pages on the Web, extract hypertext links on those pages, and automatically index the information they find to build a database. Each search engine has its own set of rules that determine how documents are found and processed. Some follow each link on each page they find, and then in turn examine each link on each of the new pages, and so on. Some people ignore the links that lead to graphics and sound files, animation files; others ignore references to resources such as WAIS databases; others are instructed to look at the most popular pages first.

Agents are the most "intelligent" of search tools. They can do more than just search: they can even carry out transactions on your behalf. Already, they can search for specific sites and return lists of sites sorted by their traffic. Agents can process the content of documents, find and index other types of resources, not just pages. They can also be programmed to extract information from pre-existing databases. Whatever information the agents index, they pass it back to the search engine database.

The general search for information on the Web is carried out by programs known as spiders. The spiders report the content of the found document, index it, and extract the resulting information. They also look at the titles, some of the links, and send the indexed information to the search engine's database.

Crawlers look at headers and only return the first link.

Robots can be programmed to follow different links of different nesting depths, perform indexing, and even check links in a document. Due to their nature, they can get stuck in cycles, so they require significant Web resources to follow links, however, there are methods designed to prevent robots from searching sites whose owners do not want them to be indexed.

Agents fetch and index different kinds information. Some, for example, index every single word in an encounter document, while others index only the most important 100 words in each, index the document's size and number of words, title, headings and subheadings, and so on. The type of index built determines what kind of searches can be made by the search engine and how the resulting information will be interpreted.

Agents can also surf the Internet and find information and then put it into a search engine database. Search engine administrators can determine which sites or types of sites agents should visit and index. The indexed information is sent to the search engine database in the same way as described above.

People can put information directly into the index by filling out a special form for the section in which they would like to put their information. This data is passed to the database.

When someone wants to find information available on the Internet, he visits a search engine page and fills out a form detailing the information he needs. Keywords, dates, and other criteria can be used here. The criteria in the search form must match the criteria used by agents when indexing the information they find while navigating the Web.

The database looks up the subject of the request based on the information provided in the completed form and outputs the corresponding documents prepared by the database. To determine the order in which the list of documents will be shown, the database uses a ranking algorithm. Ideally, the documents most relevant to the user's query will be placed first in the list. Different search engines use different ranking algorithms, but the basic principles for determining relevance are as follows:

The number of query words in the text content of the document (i.e. in the html code).

Tags in which these words are located.

The location of the search words in the document.

The share of words with respect to which relevance is determined in the total number of words in the document.

These principles apply to all search engines. And the ones below are used by some, but quite well-known (like AltaVista, HotBot).

Time - how long the page is in the search engine database. At first glance, this seems like a rather nonsensical principle. But, if you think about it, how many sites exist on the Internet that live for a maximum of a month! If the site has been around for a long time, this means that the owner is very experienced in this topic and the user is more suitable for a site that has been broadcasting to the world about the rules of conduct at the table for a couple of years than one that appeared a week ago with the same topic.

Citation index - how many links to this page lead from other pages registered in the search engine database.

3. Comparative review of reference and search systems

3.1 Gopher

Gopher was widespread on the Internet and was the forerunner of the World Wide Web. According to some reports, until 1995, Gopher was the fastest growing technology on the Internet. The growth rate of the number of eligible servers outpaced that of all other types of servers. In 1993, there were more than one and a half thousand gopher servers in the world. In fact, it was a system for distributed search and transmission of documents at the same time. Moreover, these features were not implemented as additional add-on services, like modern search engines, but were built into the system as its basic functions.

With the help of a special Veronica program, a search was carried out directly in the Gopher system using a special query language built on keywords. This system worked not only long before the advent of GOPHER (RFC-1436) is a system for searching and delivering documents stored in distributed depositories. The system was developed at the University of Minnesota (the coat of arms of this state depicts a hamster, in English gopher). The Gopher program presents the user with a sequence of menus from which they can select a topic or article of interest. The search object can be a text or a binary file (in many depositories even text files are stored in an archived, and therefore binary form), a graphic or sound image. Gopher also offers gateways to other search engines such as WWW, Wais, Archie, Whois, and network utilities such as telnet or FTP. Gopher can offer more directory convenience than FTP. For access to global network Gopher uses a client-server model. The Gopher system is now outdated, many of its servers are integrated into WEB network. But gopher was the prototype of modern WWW interfaces, and that's what makes it interesting.

3.2 WAIS

WAIS is one of the most sophisticated search engines on the Internet. It does not implement only search by fuzzy sets and probabilistic search. Unlike many search engines, the system allows you to build not only nested Boolean queries, calculate formal relevance by various proximity measures, weight query and document terms, but also correct the query by relevance. The system also allows you to use term truncation, splitting documents into fields, and maintaining distributed indexes. It is no coincidence that this system was chosen as the main search engine for the implementation of the Britannica Encyclopedia on the Internet.

The distributed information system WAIS was conceived as a network analogue of traditional information retrieval systems (IRS), allowing network users to search in full-text databases using the information retrieval language traditional for IS, the search prescriptions of which are based on keywords and/or their truncations. , interconnected by logical operators 0R or AND.

Initially, the WAIS system was developed by four firms: Dow Jones and Co. (business databases); Think Machines Corporation (information retrieval systems); Apple Computer (user interface) and KPMG Peat Maverick (work with a large number of users). The first prototype of WAIS was a semi-commercial semi-research system with severe restrictions on use by both users and database administrators. The WAIS prototype understood the natural English language and translated it into the system's search prescriptions. In reality, WAIS only became widely used with the advent of the FreeWAIS version for UNIX operating systems. Today there is a large number of implementations of WAIS, mainly commercial, and the system has become a kind of standard information retrieval engine on the Internet.

When working with WAIS, users do not have to spend a lot of time to find the materials they need.

There are more than 300 WAIS libraries on the Internet. But since the information is presented mainly by volunteers from academic organizations, most of the material is in the field of research and computer science.

3.3 WWW

WWW is a system for working with hypertext. It is potentially the most powerful search tool. Hypertext connects different documents based on a predefined set of words. For example, when a new word or concept is encountered in a text, a hypertext system makes it possible to navigate to another document in which that word or concept is discussed in more detail. Often used as an interface to WAIS databases, but the lack of hypertext links limits the possibilities. WWW to simple browsing like Gopher.

The user, for his part, can use the hypertext capability of the WWW to link between his data and the WAIS and WWW data in such a way that the user's own records are somehow integrated into the information for public access. In fact, this, of course, does not happen, but it is perceived that way.

3.4 AltaVista

Indexing in this system is carried out using a robot. In this case, the robot has the following priorities:

key phrases at the top of the page;

key phrases by the number of occurrences/presence of words/phrases;

If there are no tags on the page, it uses the first 30 words that it indexes and shows instead of a description (tag description)

Most interesting opportunity AltaVista is an advanced search. Here it is worth mentioning right away that, unlike many other systems, AltaVista supports a single NOT operator. In addition, there is also the NEAR operator, which implements the possibility of contextual search, when the terms should be located side by side in the text of the document. AltaVista allows searching by key phrases, while it has a fairly large phraseological dictionary. Among other things, when searching in AltaVista, you can specify the name of the field where the word should occur: hypertext link, applet, image name, title, and a number of other fields. Unfortunately, the ranking procedure is not described in detail in the system documentation, but it can be seen that the ranking is applied both in a simple search and in an extended query. In fact, this system can be attributed to a system with an extended boolean search.

3.5 Opentext

The OpenText information system is the most commercialized information product online. All descriptions are more like an advertisement than an informative job guide. The system allows you to search using logical connectors, but the query size is limited to three terms or phrases. In this case, we are talking about advanced search. When issuing the results, the degree of compliance of the document with the request and the size of the document are reported. The system also allows you to improve search results in the style of a traditional boolean search. OpenText could be classified as a traditional information retrieval system if it were not for the ranking mechanism.

3.6 Infoseek

In this system, the index is created by a robot, but it does not index the entire site, but only the specified page. In this case, the robot has the following priorities:

words in the title