Robots txt Яндекса

Robots txt самого Яндекса

Сегодня пятничным вечером, размышляя о вечном, решил заглянуть в robots.txt самого Яндекса и был немало удивлен, во-первых, его размером, во-вторых, пренебрежением стандартами составления роботса, изложенными у самого же Яндекса:  https://yandex.ru/support/webmaster/controlling-robot/robots-txt.xml#allow-disallow

В частности, абсолютно проигнорировано вот это примечание:

Недопустимо наличие пустых переводов строки между директивами User-agent, Disallowи Allow.

Понятно, что любой вебмастер — это человек, а человеку свойственно делать ошибки, даже если он работает в Яндексе.

И все же, для меня в этом роботсе более интересными оказались не возможные ошибки, а перечисление большого количества директорий, многие из которых, на самом деле, уже давно не существуют.
Вот так вот — директорий уже нет, а память о них еще жива в роботсе Яндекса. Получается, такая своеобразная «книга памяти». Посмотрел я на все эти длинные списки существовавших когда-то разделов, и подумал, что неплохо было бы сохранить все это на память для потомков.

Ниже идет содержимое файла robots.txt по состоянию на 01.04.2017

Если Вам известны примеры таких же больших, или еще больших файлов роботс, поделитесь ссылкой в комментариях — было бы интересно на них посмотреть.

# yandex.ru
User-agent: *
Disallow: /?
Disallow: /403.html
Disallow: /404.html
Disallow: /500.html
Disallow: /about.html
Disallow: /adddata
Disallow: /adresa-segmentator
Disallow: /advanced_engl.html
Disallow: /advertising
Disallow: /all-supported-params
Disallow: /articles
Disallow: /blog/*?text=
Disallow: /blog/*?tag=
Disallow: /blog/*/*/*
Disallow: /blogs*
Allow: /blogs$
Allow: /blogs/$
Allow: /blogs/pad/$
Allow: /blogs/pad$

Disallow: /bus/orders*
Disallow: /bus/api/*

Disallow: /catalog/?text=
Disallow: /cgi-bin/
Disallow: /cgi/
Disallow: /chisla.html
Disallow: /clck
Disallow: /cy
Disallow: /cycounter
Disallow: /dzen
Disallow: /edu/ping
Disallow: /edu/tasks
Disallow: /edu/teachers
Disallow: /edu/test
Disallow: /experiments.xml
Disallow: /formfeedback
Disallow: /goto_issue/
Disallow: /goto_rubric/
Disallow: /i/
Disallow: /images-data
Disallow: /images.html
Disallow: /images/*
Allow: /images/$
Allow: /images/smart/$
Allow: /images/touch/$
Allow: /images/touch/?
Disallow: /index_m
Disallow: /infected
Disallow: /keyboard_qwerty.html
Disallow: /logotypes
Disallow: /more_samples
Disallow: /msearch
Disallow: /msearchpart
Disallow: /map-constructor/loader*
Disallow: /maps/*?
Allow: /maps/*?lang=ru$
Allow: /maps/*?lang=uk$
Disallow: /norobot
Disallow: /opensearch.xml
Disallow: /padsearch
Disallow: /people/*
Allow: /people$
Allow: /people/$
Allow: /people/pad/$
Allow: /people/pad$
Disallow: /person
Disallow: /podpiska/login.pl
Disallow: /pogoda/search
Disallow: /pogoda/404
Disallow: /pogoda/*?*lat=*&*lon=*
Disallow: /pogoda/*?*lon=*&*lat=*

Disallow: /polling
Disallow: /poll/
Disallow: /promo/*goodbye*
Disallow: /promo/diskelement
Disallow: /promo/launcher/mgoodbye/*
Disallow: /promo/skype*?*
Disallow: /promo/skype/*/*
Disallow: /promo/*welcome*
Disallow: /promo/yobject/changelog*
Disallow: /soft/bm/goodbye
Disallow: /soft/chrome/ext-install/*
Disallow: /soft/chrome/searchline-install/*
Disallow: /soft/*goodbye*
Disallow: /soft/punto/mac/uninstall/*
Disallow: /soft/punto/win/uninstall/*
Disallow: /soft/*welcome*
Disallow: /soft/win/?*
Disallow: /quotes
Disallow: /redir
Disallow: /region_map
Disallow: /regions_list.xml
Disallow: /regions.html?
Disallow: /rubric2sport
Disallow: /s/
Disallow: /save
Disallow: /search
Allow: /search/advanced
Disallow: /setup
Disallow: /showcaptcha
Disallow: /sitesearch
Disallow: /skazki
Disallow: /sl/*.html
Disallow: /soft/extensions/goodbye
Disallow: /sportagent
Disallow: /storeclick
Disallow: /storerequest
Disallow: /subscribe/confirm.pl
Disallow: /subscribe/view.pl

Allow: /support/
Disallow: /support/direct-images
Disallow: /support/direct-tooltips
Disallow: /support/distr
Disallow: /support/dsp
Disallow: /support/fe4be44a295cc679e19bf0b8f133083d
Disallow: /support/maps-beta
Disallow: /support/market-images
Disallow: /support/security
Disallow: /support/webmaster-images
Disallow: /support/*zout_
Disallow: /support/search-results/

Disallow: /telsearch
Disallow: /themes
Disallow: /toggle-experiment
Disallow: /touchsearch
Disallow: /versions
Disallow: /video/*
Disallow: /video/*filmId=*
Allow: /video/$
Allow: /video/?
Allow: /video/sitemap
Allow: /video/%D0%B7%D0%B0%D0%BF%D1%80%D0%BE%D1%81/
Allow: /video/dizi-izle/
Allow: /video/dizi-izle/?
Allow: /video/yerli-dizi-izle/
Allow: /video/yabanci-dizi-izle/
Allow: /video/touch/
Allow: /video/pad/
Disallow: /xmlsearch

Disallow: /yaca
Allow: /yaca/$
Allow: /yaca/m/$
Allow: /yaca/m/yca/cat/*/$
Allow: /yaca/*.xml$
Allow: /yaca/cat/*/$
Disallow: /yaca/result.xml
Disallow: /yaca/*/tungrp/
Disallow: /yaca/*/time/
Disallow: /yaca/*/geo/
Disallow: /yaca/cat/*/geo/*
Disallow: /yaca/*/synt2/
Disallow: /yandsearch
Disallow: /yca/cy

Allow: /yac2014
Disallow: /soft/distribution
Disallow: /soft/?*
Disallow: /promo/launcher/?*
Disallow: /opera/?*
Disallow: /firefox/?*
Disallow: /soft/bm/?*
Disallow: /soft/browsers/?*
Disallow: /soft/punto/?*
Disallow: /ie/?*
Disallow: /element/?*
Disallow: /element/*goodbye*
Disallow: /soft/*goodbye*
Disallow: /adult

Clean-Param: ncrnd&redircnt&clid&_&win /

Sitemap: https://yandex.ru/pogoda/sitemap_index.xml
Sitemap: https://yandex.ru/support/sitemap.xml
Sitemap: https://yandex.ru/blog/sitemap.xml

User-Agent: Twitterbot
Allow: /images
Allow: /gorsel
Allow: /video
Disallow: /adult
Sitemap: https://yandex.ru/pogoda/sitemap_index.xml
Sitemap: https://yandex.ru/support/sitemap.xml
Sitemap: https://yandex.ru/blog/sitemap.xml

User-agent: Applebot
Disallow: /?
Disallow: /403.html
Disallow: /404.html
Disallow: /500.html
Disallow: /about.html
Disallow: /adddata
Disallow: /adresa-segmentator
Disallow: /advanced_engl.html
Disallow: /advertising
Disallow: /all-supported-params
Disallow: /articles
Disallow: /blog/*?text=
Disallow: /blog/*?tag=
Disallow: /blog/*/*/*
Disallow: /blogs*
Allow: /blogs$
Allow: /blogs/$
Allow: /blogs/pad/$
Allow: /blogs/pad$

Disallow: /bus/orders*
Disallow: /bus/api/*

Disallow: /catalog/?text=
Disallow: /cgi-bin/
Disallow: /cgi/
Disallow: /chisla.html
Disallow: /clck
Disallow: /cy
Disallow: /cycounter
Disallow: /dzen
Disallow: /edu/ping
Disallow: /edu/tasks
Disallow: /edu/teachers
Disallow: /edu/test
Disallow: /experiments.xml
Disallow: /formfeedback
Disallow: /goto_issue/
Disallow: /goto_rubric/
Disallow: /i/
Disallow: /images-data
Disallow: /images.html
Disallow: /images/*
Allow: /images/$
Allow: /images/smart/$
Allow: /images/touch/$
Allow: /images/touch/?
Disallow: /index_m
Disallow: /infected
Disallow: /keyboard_qwerty.html
Disallow: /logotypes
Disallow: /map-constructor/loader*
Disallow: /more_samples
Disallow: /msearch
Disallow: /msearchpart
Disallow: /norobot
Disallow: /opensearch.xml
Disallow: /padsearch
Disallow: /people/*
Allow: /people$
Allow: /people/$
Allow: /people/pad/$
Allow: /people/pad$
Disallow: /person
Disallow: /podpiska/login.pl
Disallow: /pogoda/search
Disallow: /pogoda/404
Disallow: /pogoda/*?*lat=*&*lon=*
Disallow: /pogoda/*?*lon=*&*lat=*

Disallow: /polling
Disallow: /poll/
Disallow: /promo/*goodbye*
Disallow: /promo/diskelement
Disallow: /promo/launcher/mgoodbye/*
Disallow: /promo/skype*?*
Disallow: /promo/skype/*/*
Disallow: /promo/*welcome*
Disallow: /promo/yobject/changelog*
Disallow: /soft/bm/goodbye
Disallow: /soft/chrome/ext-install/*
Disallow: /soft/chrome/searchline-install/*
Disallow: /soft/*goodbye*
Disallow: /soft/punto/mac/uninstall/*
Disallow: /soft/punto/win/uninstall/*
Disallow: /soft/*welcome*
Disallow: /soft/win/?*
Disallow: /quotes
Disallow: /redir
Disallow: /region_map
Disallow: /regions_list.xml
Disallow: /regions.html?
Disallow: /rubric2sport
Disallow: /s/
Disallow: /save
Disallow: /search
Allow: /search/advanced
Disallow: /set/
Disallow: /setup
Disallow: /showcaptcha
Disallow: /sitesearch
Disallow: /skazki
Disallow: /sl/*.html
Disallow: /soft/extensions/goodbye
Disallow: /sportagent
Disallow: /storeclick
Disallow: /storerequest
Disallow: /subscribe/confirm.pl
Disallow: /subscribe/view.pl

Allow: /support/
Disallow: /support/direct-images
Disallow: /support/direct-tooltips
Disallow: /support/distr
Disallow: /support/dsp
Disallow: /support/fe4be44a295cc679e19bf0b8f133083d
Disallow: /support/maps-beta
Disallow: /support/market-images
Disallow: /support/security
Disallow: /support/webmaster-images
Disallow: /support/*zout_
Disallow: /support/search-results/

Disallow: /telsearch
Disallow: /themes
Disallow: /toggle-experiment
Disallow: /touchsearch
Disallow: /versions
Disallow: /video/*
Disallow: /video/*filmId=*
Allow: /video/$
Allow: /video/?
Allow: /video/sitemap
Allow: /video/%D0%B7%D0%B0%D0%BF%D1%80%D0%BE%D1%81/
Allow: /video/dizi-izle/
Allow: /video/dizi-izle/?
Allow: /video/yerli-dizi-izle/
Allow: /video/yabanci-dizi-izle/
Allow: /video/touch/
Allow: /video/pad/
Disallow: /xmlsearch

Disallow: /yaca
Allow: /yaca/$
Allow: /yaca/m/$
Allow: /yaca/m/yca/cat/*/$
Allow: /yaca/*.xml$
Allow: /yaca/cat/*/$
Disallow: /yaca/result.xml
Disallow: /yaca/*/tungrp/
Disallow: /yaca/*/time/
Disallow: /yaca/*/geo/
Disallow: /yaca/cat/*/geo/*
Disallow: /yaca/*/synt2/
Disallow: /yandsearch
Disallow: /yca/cy

Allow: /yac2014
Disallow: /soft/distribution
Disallow: /soft/?*
Disallow: /promo/launcher/?*
Disallow: /opera/?*
Disallow: /firefox/?*
Disallow: /soft/bm/?*
Disallow: /soft/browsers/?*
Disallow: /soft/punto/?*
Disallow: /ie/?*
Disallow: /element/?*
Disallow: /element/*goodbye*
Disallow: /soft/*goodbye*
Disallow: /adult

Clean-Param: ncrnd&redircnt&clid&_&win /

Sitemap: https://yandex.ru/pogoda/sitemap_index.xml
Sitemap: https://yandex.ru/support/sitemap.xml
Sitemap: https://yandex.ru/blog/sitemap.xml

User-agent: Yandex
Disallow: /?
Disallow: /403.html
Disallow: /404.html
Disallow: /500.html
Disallow: /about.html
Disallow: /adddata
Disallow: /adresa-segmentator
Disallow: /advanced_engl.html
Disallow: /advertising
Disallow: /all-supported-params
Disallow: /articles
Disallow: /blog/*?text=
Disallow: /blog/*?tag=
Disallow: /blog/*/*/*
Disallow: /blogs*
Allow: /blogs$
Allow: /blogs/$
Allow: /blogs/pad/$
Allow: /blogs/pad$

Disallow: /bus/orders*
Disallow: /bus/api/*

Disallow: /catalog/?text=
Disallow: /cgi-bin/
Disallow: /cgi/
Disallow: /chisla.html
Disallow: /clck
Disallow: /cy
Disallow: /cycounter
Disallow: /dzen
Disallow: /edu/ping
Disallow: /edu/tasks
Disallow: /edu/teachers
Disallow: /edu/test
Disallow: /experiments.xml
Disallow: /formfeedback
Disallow: /goto_issue/
Disallow: /goto_rubric/
Disallow: /i/
Disallow: /images-data
Disallow: /images.html
Disallow: /images/*
Allow: /images/$
Allow: /images/smart/$
Allow: /images/touch/$
Allow: /images/touch/?
Disallow: /index_m
Disallow: /infected
Disallow: /keyboard_qwerty.html
Disallow: /logotypes
Disallow: /more_samples
Disallow: /msearch
Disallow: /msearchpart
Disallow: /map-constructor/loader*
Disallow: /maps/*?
Allow: /maps/*?lang=ru$
Allow: /maps/*?lang=uk$
Disallow: /norobot
Disallow: /opensearch.xml
Disallow: /padsearch
Disallow: /people/*
Allow: /people$
Allow: /people/$
Allow: /people/pad/$
Allow: /people/pad$
Disallow: /person
Disallow: /podpiska/login.pl
Disallow: /pogoda/search
Disallow: /pogoda/404
Disallow: /pogoda/*?*lat=*&*lon=*
Disallow: /pogoda/*?*lon=*&*lat=*

Disallow: /polling
Disallow: /poll/
Disallow: /promo/*goodbye*
Disallow: /promo/diskelement
Disallow: /promo/launcher/mgoodbye/*
Disallow: /promo/skype*?*
Disallow: /promo/skype/*/*
Disallow: /promo/*welcome*
Disallow: /promo/yobject/changelog*
Disallow: /soft/bm/goodbye
Disallow: /soft/chrome/ext-install/*
Disallow: /soft/chrome/searchline-install/*
Disallow: /soft/*goodbye*
Disallow: /soft/punto/mac/uninstall/*
Disallow: /soft/punto/win/uninstall/*
Disallow: /soft/*welcome*
Disallow: /soft/win/?*
Disallow: /quotes
Disallow: /redir
Disallow: /region_map
Disallow: /regions_list.xml
Disallow: /regions.html?
Disallow: /rubric2sport
Disallow: /s/
Disallow: /save
Disallow: /search
Allow: /search/advanced
Disallow: /setup
Disallow: /showcaptcha
Disallow: /sitesearch
Disallow: /skazki
Disallow: /sl/*.html
Disallow: /soft/extensions/goodbye
Disallow: /sportagent
Disallow: /storeclick
Disallow: /storerequest
Disallow: /subscribe/confirm.pl
Disallow: /subscribe/view.pl

Allow: /support/
Disallow: /support/direct-images
Disallow: /support/direct-tooltips
Disallow: /support/distr
Disallow: /support/dsp
Disallow: /support/fe4be44a295cc679e19bf0b8f133083d
Disallow: /support/maps-beta
Disallow: /support/market-images
Disallow: /support/security
Disallow: /support/webmaster-images
Disallow: /support/*zout_
Disallow: /support/search-results/

Disallow: /telsearch
Disallow: /themes
Disallow: /toggle-experiment
Disallow: /touchsearch
Disallow: /versions
Disallow: /video/*
Disallow: /video/*filmId=*
Disallow: /video/%D0%B7%D0%B0%D0%BF%D1%80%D0%BE%D1%81/%D1%81%D0%B5%D1%80%D0%B8%D0%B0%D0%BB/*
Allow: /video/$
Allow: /video/?
Allow: /video/sitemap
Allow: /video/%D0%B7%D0%B0%D0%BF%D1%80%D0%BE%D1%81/
Allow: /video/dizi-izle/
Allow: /video/dizi-izle/?
Allow: /video/yerli-dizi-izle/
Allow: /video/yabanci-dizi-izle/
Allow: /video/touch/
Allow: /video/pad/
Disallow: /xmlsearch

Disallow: /yaca
Allow: /yaca/$
Allow: /yaca/m/$
Allow: /yaca/m/yca/cat/*/$
Allow: /yaca/*.xml$
Allow: /yaca/cat/*/$
Disallow: /yaca/result.xml
Disallow: /yaca/*/tungrp/
Disallow: /yaca/*/time/
Disallow: /yaca/*/geo/
Disallow: /yaca/cat/*/geo/*
Disallow: /yaca/*/synt2/
Disallow: /yandsearch
Disallow: /yca/cy

Allow: /yac2014
Disallow: /soft/distribution
Disallow: /soft/?*
Disallow: /promo/launcher/?*
Disallow: /opera/?*
Disallow: /firefox/?*
Disallow: /soft/bm/?*
Disallow: /soft/browsers/?*
Disallow: /soft/punto/?*
Disallow: /ie/?*
Disallow: /element/?*
Disallow: /element/*goodbye*
Disallow: /soft/*goodbye*
Disallow: /adult

Clean-Param: ncrnd&redircnt&clid&_&win /

Понравилась статья? Поделиться с друзьями:
Добавить комментарий

;-) :| :x :twisted: :smile: :shock: :sad: :roll: :razz: :oops: :o :mrgreen: :lol: :idea: :grin: :evil: :cry: :cool: :arrow: :???: :?: :!: