Server Fun

FYI, if you noticed and were wondering why my various sites have been down so much … there is a story.

I noticed lots of reboots starting this afternoon, and this evening I found that the site would only stay up for 1 minute at a time. It would reboot so much that Linode would stop trying and leave it powered off.

I filed a support ticket and then tried various rescue mode things. I fsck the disk, I did the whole chroot thing and updated all packages. The tech handling my ticket noticed the console.log was full of apache2 processes spinning up, running the VM out of memory and then kernel panicking. I configured apache2 to allow for a lower number of max processes.

When I did that, the server stayed up long enough for me to get to my home directory and do a multi-tail on every access log of every site on there. When I did, I noticed that one obscure site was getting hammered on its xmlrpc.php, to the tune of multiple hits every second from three IP addresses. This was the culprit. Three iptables DROP rules later, all this nonsense was gone.

This wasn’t the most fun evening I’ve ever had (and this shit consumed the ENTIRE evening) but it probably has a positive outcome. This Linode server has always rebooted more than I like, daily at times but is seldom up more than 2 solid weeks. That has probably always been a problem with the apache2 default configuration being more than this small VM can handle. By configuring it to have a lower max usage, that may solve the rebooting issue.

Also on: