Webcrawler
Robot Abuse
What the hell happened to netiquette in the creation of web crawling robots? I seem to remember 30 seconds being the absolute minimum interval between requests to the same domain for a well-behaved robot, and several minutes being even better. Jesus, the crawler I wrote for a project in grad school...
read article