Server Fun

FYI, if you noticed and were wondering why my various sites have been down so much … there is a story.

I noticed lots of reboots starting this afternoon, and this evening I found that the site would only stay up for 1 minute at a time. It would reboot so much that Linode would stop trying and leave it powered off.

I filed a support ticket and then tried various rescue mode things. I fsck the disk, I did the whole chroot thing and updated all packages. The tech handling my ticket noticed the console.log was full of apache2 processes spinning up, running the VM out of memory and then kernel panicking. I configured apache2 to allow for a lower number of max processes.

When I did that, the server stayed up long enough for me to get to my home directory and do a multi-tail on every access log of every site on there. When I did, I noticed that one obscure site was getting hammered on its xmlrpc.php, to the tune of multiple hits every second from three IP addresses. This was the culprit. Three iptables DROP rules later, all this nonsense was gone.

This wasn’t the most fun evening I’ve ever had (and this shit consumed the ENTIRE evening) but it probably has a positive outcome. This Linode server has always rebooted more than I like, daily at times but is seldom up more than 2 solid weeks. That has probably always been a problem with the apache2 default configuration being more than this small VM can handle. By configuring it to have a lower max usage, that may solve the rebooting issue.

Also on:

Evil Genius Chronicles Podcast for September 28 2015 – New Everything

In this episode, I play a song from Shilelagh Law; I talk about my training course I taught; I discuss dividing my work and life laptops better, creating a standing desk and organizing my space better; I talk about converting my podcasting life into small bits of work every day; I talk more about how podcasts and blogs can lose me and how shows that take extra work reduce their value to me.

Here is the direct MP3 download for the Evil Genius Chronicles podcast, September 28 2015

Links mentioned in this episode:

You can subscribe to this podcast feed via RSS. To sponsor the show, contact BackBeat Media. Don’t forget, you can fly your EGC flag by buying the stuff package. This show as a whole is Creative Commons licensed Attribution-NonCommercial 3.0 Unported. Bandwidth for this episode is provided by Cachefly.

Cohost with Me

Looking for co-hosts for a monthly podcast about Fraction and DeConnick’s work. First show next weekend. Reach out if interested.

Also on:

One More Thing

Faces on the Street in Mumbai

I don’t know if I made it clear, the important thing about the training course I taught last week is that it gated my internal transfer. With it done, I’m now at work as a developer evangelist/solution architect. My new boss and I worked out a list of my goals for Q4 yesterday. When it was done, he asked me how I felt about the list. I said, “Frankly, I feel bad taking money to do all this.”

Photo by Willem Velthoven

Also on:

First Things First

It was not the most important thing on my plate, but I took 10 minutes anyway to update my position on LinkedIn. This was purely for mental health reasons.

Also on:

Final Tally

I spent five days in San Diego at the main office, preparing for and then conducting our training session. Basically, as my partner and I move on to other roles in the company, we trained up a squad of consultants and partners to do the work he had been doing.

The training was well received, with no one in the room feeling unprepared to face the future. I’ve seen complimentary emails moving about, so both higher-ups and the trainees seem happy with the whole deal.

On top of that, I was prepared to take a weight hit, hoping to confine it to a few pounds. Even though I still ate salad at the cafeteria every day and only budgeted one single meal (at Eureka Burger, of course) to go crazy, it seemed possible that just traveling and eating airport food, higher beer consumption, etc might catch up to me. In fact, this morning I was half a pound under where I was last week.

So, all told it was best case scenario all around.

Also on:

Evil Genius Chronicles Podcast for September 19 2015 – Training on the Road

In this episode recorded in a hotel room, I talk about my business trip and my first ever stint giving corporate training.

Here is the direct MP3 download for the Evil Genius Chronicles podcast, September 19 2015.

Links mentioned in this episode:

You can subscribe to this podcast feed via RSS. To sponsor the show, contact BackBeat Media. Don’t forget, you can fly your EGC flag by buying the stuff package. This show as a whole is Creative Commons licensed Attribution-NonCommercial 3.0 Unported. Bandwidth for this episode is provided by Cachefly.

Also on:

End of Training

Training ends in a few hours. By all reasonable metrics, it was rousing success. I enjoyed it but am tired and ready to go home.

Also on:

Class Day Two

Gearing up for day two of the course. Funny thing, it gets real easy for me now. Most of the talking is done. From here out, they mostly work labs. The are designed to be challenging, so there will be a lot of interaction, but the solid multi-hour blocks of me with Powerpoint are over.

Writing these labs and creating the special version of our product with just enough problems to make this challenging for them was laborious and like pulling teeth. Conducting the lab by comparison should be a cake walk.

Also on:

Go Time

Been thinking about this training course for months, writing it for weeks. Today is the day, go time. I am excited and nervous.

Also on:

Evil Genius Chronicles Podcast for September 10 2015 – DDOP 36, Extra Innings

In this episode, I answer the final batch of questions from Reed Knight and close out the Dog Days of Podcasting challenge for 2015.

Here is the direct MP3 download for the Evil Genius Chronicles podcast, September 10 2015

Links mentioned in this episode:

You can subscribe to this podcast feed via RSS. To sponsor the show, contact BackBeat Media. Don’t forget, you can fly your EGC flag by buying the stuff package. This show as a whole is Creative Commons licensed Attribution-NonCommercial 3.0 Unported. Bandwidth for this episode is provided by Cachefly.

Also on: