SearchTools Blog (searchtools) wrote,
SearchTools Blog

New Robot Exclusion Protocol!

Supported by webwide search engines Yahoo, Google and Microsoft, this adds directives to robots.txt:
  • "Allow" directives
  • wildcards in URLs
  • Sitemap Location
There are also HTML meta tags and document properties directives for
  • NOODP (don't use ODP information for this page).
Yahoo has a nice long blog entry on this, as does Google and MS Live Search. Great news for web developers, who've been waiting for this for a very long time.

But there's nothing from the robots mailing list or the which is a shame.

This is also a test for all site and intranet search crawlers -- any abandoned software will not recognize these new directives.

I'll dig further into this in the next week and provide more analysis and details.


  • Post a new comment


    default userpic

    Your reply will be screened

    Your IP address will be recorded 

    When you submit the form an invisible reCAPTCHA check will be performed.
    You must follow the Privacy Policy and Google Terms of use.