Server Fun

FYI, if you noticed and were wondering why my various sites have been down so much … there is a story.

I noticed lots of reboots starting this afternoon, and this evening I found that the site would only stay up for 1 minute at a time. It would reboot so much that Linode would stop trying and leave it powered off.

I filed a support ticket and then tried various rescue mode things. I fsck the disk, I did the whole chroot thing and updated all packages. The tech handling my ticket noticed the console.log was full of apache2 processes spinning up, running the VM out of memory and then kernel panicking. I configured apache2 to allow for a lower number of max processes.

When I did that, the server stayed up long enough for me to get to my home directory and do a multi-tail on every access log of every site on there. When I did, I noticed that one obscure site was getting hammered on its xmlrpc.php, to the tune of multiple hits every second from three IP addresses. This was the culprit. Three iptables DROP rules later, all this nonsense was gone.

This wasn’t the most fun evening I’ve ever had (and this shit consumed the ENTIRE evening) but it probably has a positive outcome. This Linode server has always rebooted more than I like, daily at times but is seldom up more than 2 solid weeks. That has probably always been a problem with the apache2 default configuration being more than this small VM can handle. By configuring it to have a lower max usage, that may solve the rebooting issue.

Published by

dave

Dave Slusher is a blogger, podcaster, computer programmer, author, science fiction fan and father. Member of the Podcast Hall of Fame class of 2022.

9 thoughts on “Server Fun”

  1. Turns out the problem last night was just the tip of the spear. More IPs are now hitting more sites. I added the iptables-persistent package to preserve the DROP rules through reboot and have a bash script mailing me every 30 minutes with the uptime and new hits to xmlrpc.php. This is not how I want to spend my time, in an arms race with griefers.

    1. I have installed xmlrpc.php blockers on sites that don’t need, but that doesn’t solve the problem of them still hitting apache and making it spin processes. I do think Jetpack from WordPress uses it as well, so I am leaving it up on EGC. That site wasn’t the problem (yet). I was being brought down by traffic on http://APigLikeThat.com . Hoist on my own petard.

  2. J Edward Wynia The tech thought he was just making an aside but probably saved me hours by pointing me to apache2 config. Otherwise I would have been focusing on the reboots for a very long time when they were just a symptom.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.