Tag: my glamorous life

Server Fun

FYI, if you noticed and were wondering why my various sites have been down so much … there is a story.

I noticed lots of reboots starting this afternoon, and this evening I found that the site would only stay up for 1 minute at a time. It would reboot so much that Linode would stop trying and leave it powered off.

I filed a support ticket and then tried various rescue mode things. I fsck the disk, I did the whole chroot thing and updated all packages. The tech handling my ticket noticed the console.log was full of apache2 processes spinning up, running the VM out of memory and then kernel panicking. I configured apache2 to allow for a lower number of max processes.

When I did that, the server stayed up long enough for me to get to my home directory and do a multi-tail on every access log of every site on there. When I did, I noticed that one obscure site was getting hammered on its xmlrpc.php, to the tune of multiple hits every second from three IP addresses. This was the culprit. Three iptables DROP rules later, all this nonsense was gone.

This wasn’t the most fun evening I’ve ever had (and this shit consumed the ENTIRE evening) but it probably has a positive outcome. This Linode server has always rebooted more than I like, daily at times but is seldom up more than 2 solid weeks. That has probably always been a problem with the apache2 default configuration being more than this small VM can handle. By configuring it to have a lower max usage, that may solve the rebooting issue.

Class Day Two

Gearing up for day two of the course. Funny thing, it gets real easy for me now. Most of the talking is done. From here out, they mostly work labs. The are designed to be challenging, so there will be a lot of interaction, but the solid multi-hour blocks of me with Powerpoint are over.

Writing these labs and creating the special version of our product with just enough problems to make this challenging for them was laborious and like pulling teeth. Conducting the lab by comparison should be a cake walk.