Robotstxt
Crawl-Delay in Robots.txt
Somehow or another, I've never noticed the extensions to the robots.txt file. I've grown fed up with crawlers that hit my blog 45 times a minute, which can be quite debilitating when five different ones are all hitting me at the same time. Although it seems like the "official" keepers of...
read article