Prohibiting indexing robots txt. Hiding links with scripts

The purpose of this guide is to help webmasters and administrators use robots.txt.

Introduction

The Robot Exception Standard is inherently very simple. In short, it works like this:

When a compliant robot visits a site, it first requests a file called “/robots.txt”. If such a file is found, the Robot looks for instructions in it that prohibit indexing some parts of the site.

Where to put your robots.txt file

The robot simply requests the URL "/robots.txt" on your site, the site in in this case Is a specific host on a specific port.

There can be only one "/robots.txt" file on a site. For example, you shouldn't put your robots.txt file in custom subdirectories - anyway, robots won't look for them there. If you want to be able to create robots.txt files in subdirectories, then you need a way to programmatically collect them into a single robots.txt file located at the root of your site. You can use instead.

Remember that URLs are case sensitive and the file name “/robots.txt” must be written entirely in lowercase.

As you can see, the robots.txt file should be placed exclusively in the site root.

What to write to your robots.txt file

The robots.txt file is usually written something like:

User-agent: *
Disallow: / cgi-bin /
Disallow: / tmp /
Disallow: / ~ joe /

In this example, indexing of three directories is prohibited.

Note that each directory is listed on a separate line - you cannot write "Disallow: / cgi-bin / / tmp /". You also cannot split one Disallow or User-agent statement into several lines, because line break is used to separate instructions from each other.

Regular expressions and wildcards cannot be used either. An asterisk (*) in a User-agent statement means any robot. Instructions like "Disallow: * .gif" or "User-agent: Ya *" are not supported.

The specific instructions in robots.txt depend on your site and what you want to block from indexing. Here are some examples:

Deny the entire site from being indexed by all robots

User-agent: *
Disallow: /

Allow all robots to index the entire site

User-agent: *
Disallow:

Or you can just create empty file"/Robots.txt".

Close only a few directories from indexing

User-agent: *
Disallow: / cgi-bin /
Disallow: / tmp /
Disallow: / private /

Disallow site indexing for only one robot

User-agent: BadBot
Disallow: /

Allow site indexing for one robot and prohibit all others

User-agent: Yandex
Disallow:

User-agent: *
Disallow: /

Deny indexing all files except one

This is not easy because there is no “Allow” statement. Instead, you can move all files except for the one you want to allow for indexing into a subdirectory and prohibit its indexing:

User-agent: *
Disallow: / docs /

Or you can deny all files prohibited for indexing:

User-agent: *
Disallow: /private.html
Disallow: /foo.html
Disallow: /bar.html

Any page on the site can be opened or closed for indexing by search engines. If the page is open, the search engine adds it to its index, if it is closed, then the robot does not visit it and does not take into account search results.

When creating a site, it is important at the programmatic level to close all pages from indexing that, for whatever reason, should not be seen by users and search engines.

These pages include the administrative part of the site (admin panel), pages with various service information (for example, with personal data of registered users), pages with multi-level forms (for example, complex shapes registration), forms feedback etc.

Example:
User profile on the forum about search engines ah Searchengines.

It is also mandatory to close the pages from indexing, the content of which is already used on other pages. Such pages are called duplicate. Full or partial duplicates greatly pessimize the site, as they increase the amount of non-unique content on the site.

As you can see, the content on both pages overlaps. Therefore, category pages on WordPress sites are closed from indexing, or only the name of the posts is displayed on them.

The same goes for tag pages - such pages are often found in the structure of WordPress blogs. The tag cloud makes it easier to navigate the site and allows users to quickly find information of interest. However, they are partial duplicates of other pages, which means they must be closed from indexing.

Another example is a store on CMS OpenCart.

Product category page http://www.masternet-instrument.ru/Lampy-energosberegajuschie-c-906_910_947.html.

The page of products covered by the discount http://www.masternet-instrument.ru/specials.php.

These pages have similar content, as they contain many of the same products.

Especially critical for duplicate content on different pages site belongs to Google. Per a large number of duplicates in Google, you can earn certain sanctions up to the temporary exclusion of the site from the search results.

Another case when the content of pages should not be "shown" to a search engine is pages with non-unique content. A typical example is the instructions for medicines in an online pharmacy. The content on the page with the description of the drug http://www.piluli.ru/product271593/product_info.html is not unique and has been published on hundreds of other sites.

It is almost impossible to make it unique, since rewriting such specific texts is a thankless and forbidden task. The best solution in this case, the page will be closed from indexing, or a letter will be written to search engines with a request to be loyal to the non-unique content of the content, which cannot be made unique for one reason or another.

How to block pages from indexing

The classic tool for closing pages from indexing is the robots.txt file. It is located in the root directory of your site and is created specifically to show search robots which pages they should not visit. This is normal text file, which you can edit at any time. If you don't have a robots.txt file, or if it's empty, search engines will index all pages they find by default.

The structure of the robots.txt file is pretty simple. It can consist of one or several blocks (instructions). Each instruction, in turn, consists of two lines. The first line is called User-agent and defines which search engine should follow this instruction. If you want to disable indexing for all search engines, the first line should look like this:

If you want to prohibit page indexing for only one search engine, for example, for Yandex, the first line looks like this:

The second line of the instruction is called Disallow. To ban all pages on the site, write the following on this line:

To enable indexing of all pages, the second line should look like this:

In the Disallow line, you can specify specific folders and files to be closed from indexing.

For example, to prohibit indexing of the images folder and all of its contents, write:

To "hide" specific files from search engines, we list them:

User-agent: *
Disallow: /myfile1.htm
Disallow: /myfile2.htm
Disallow: /myfile3.htm

These are the basic principles of robots.txt file structure. They will help you close individual pages and folders on your site from indexing.

Another, less common way of prohibiting indexing is the Robots meta tag. If you want to close a page from indexing or prevent search engines from indexing links placed on it, you need to write this tag in its HTML code. It must be placed in the HEAD area, before the tag