Crawl-Delay in Robots.txt

Somehow or another, I’ve never noticed the extensions to the robots.txt file. I’ve grown fed up with crawlers that hit my blog 45 times a minute, which can be quite debilitating when five different ones are all hitting me at the same time. Although it seems like the “official” keepers of the robots.txt psuedo-standard don’t […]