Pingstorm

It turns out that that XML-RPC ping plugin had a biiiiig flaw, one that resulted in an unintentional denial of service attack on my own weblog. The way it would work is by first doing the ping for each of the URLs defined in a list. If all were successful, it would write a “touch” file and use that timestamp to mark the point at which the last successful round of pings was sent. On loading the page subsequent times, it would check to see if the newest article is newer than that timestamp. If so, it would ping again, otherwise not. The problem came when any of the URLs failed. Because it didn’t write the touch file unless all succeeded, if one of the hosts was down it would never write the file. Since the side effect of pinging these hosts is to make them load the weblog, this quickly snowballed out of control. Every weblog load caused each of the hosts to be pinged, which caused them to load the page again. Weblogs.com was good about not reloading in this situation, but blo.gs and technorati happily reloaded after each ping, even if the last was less than a second previously. This got so out of hand that I had to shut down the webserver in order to allow the load to drop low enough for me to fix it.

I went into the code and made some changes. The touch file is no longer conditional on success of the XML-RPC pings, and it happens before any of the pings are sent. That way, a page load that happens while the group of hosts are in the process of being pinged will not cause additional pings. This seems to have short-circuited the infinite loop. I don’t really care that much if it doesn’t retry on a failed ping. There will be other tries later.