Server Fun

Posted on September 29, 2015
Filed Under weblogs | 9 Comments

FYI, if you noticed and were wondering why my various sites have been down so much … there is a story.

I noticed lots of reboots starting this afternoon, and this evening I found that the site would only stay up for 1 minute at a time. It would reboot so much that Linode would stop trying and leave it powered off.

I filed a support ticket and then tried various rescue mode things. I fsck the disk, I did the whole chroot thing and updated all packages. The tech handling my ticket noticed the console.log was full of apache2 processes spinning up, running the VM out of memory and then kernel panicking. I configured apache2 to allow for a lower number of max processes.

When I did that, the server stayed up long enough for me to get to my home directory and do a multi-tail on every access log of every site on there. When I did, I noticed that one obscure site was getting hammered on its xmlrpc.php, to the tune of multiple hits every second from three IP addresses. This was the culprit. Three iptables DROP rules later, all this nonsense was gone.

This wasn’t the most fun evening I’ve ever had (and this shit consumed the ENTIRE evening) but it probably has a positive outcome. This Linode server has always rebooted more than I like, daily at times but is seldom up more than 2 solid weeks. That has probably always been a problem with the apache2 default configuration being more than this small VM can handle. By configuring it to have a lower max usage, that may solve the rebooting issue.

Tags: , ,


There is a posted comment policy for this blog. Please respect the rules.

9 Responses to “Server Fun”

    Comment Permalink
  1. Linda Tewes on September 29th, 2015 11:58 pm

    Linda Tewes liked this Article on

  2. Comment Permalink
  3. Garrick van Buren Garrick van Buren on September 30th, 2015 9:17 am

    I recently had to deal with a very similar issue.

  4. Comment Permalink
  5. Dave Slusher Dave Slusher on September 30th, 2015 9:42 am

    Turns out the problem last night was just the tip of the spear. More IPs are now hitting more sites. I added the iptables-persistent package to preserve the DROP rules through reboot and have a bash script mailing me every 30 minutes with the uptime and new hits to xmlrpc.php. This is not how I want to spend my time, in an arms race with griefers.

  6. Comment Permalink
  7. J Edward Wynia J Edward Wynia on September 30th, 2015 9:44 am

    Could you flip the rules and whitelist your ip for access to xmlrpc.php (which I assume is for post publishing or similar purposes and not general consumption)?

  8. Comment Permalink
  9. Dave Slusher Dave Slusher on September 30th, 2015 9:46 am

    I have installed xmlrpc.php blockers on sites that don’t need, but that doesn’t solve the problem of them still hitting apache and making it spin processes. I do think Jetpack from WordPress uses it as well, so I am leaving it up on EGC. That site wasn’t the problem (yet). I was being brought down by traffic on . Hoist on my own petard.

  10. Comment Permalink
  11. PJ Cabrera PJ Cabrera on September 30th, 2015 10:29 am

    ISIS conducting cyberattacks against infidels spreading sin with fresh bacon?

  12. Comment Permalink
  13. J Edward Wynia J Edward Wynia on September 30th, 2015 9:42 am

    That’s the kind of particularly nasty problem where the problem keeps getting in the way of figuring out what the problem is.

  14. Comment Permalink
  15. Dave Slusher Dave Slusher on September 30th, 2015 9:44 am

    J Edward Wynia The tech thought he was just making an aside but probably saved me hours by pointing me to apache2 config. Otherwise I would have been focusing on the reboots for a very long time when they were just a symptom.

  16. Comment Permalink
  17. PJ Cabrera PJ Cabrera on September 30th, 2015 10:33 am

    A problem like that, you don’t fix all at once

Leave a Reply