That Sinking Feeling

On Friday of PuppetConf I jumped on a video call with my coworker back at the office to get news that the president of engineering had asked for me to join the team supporting the new site that we had recently acquired. Everything about this was super exciting. It brought with it the chance to finally do something production in AWS and the potential to be a little more focused on a single property.

While I have generally kept up with how AWS works and have dabbled a bit here and there, I have never done anything that would constitute production. I had heard about the occasional shit show that is EBS and was able to grok how all the components came together. In my excitement, I spent the weekend using Cloud Formation to spin up simple HA setups and then tying in New Relic for monitoring.

Then Monday hit. Like a ton of bricks. So many unanswered questions and so much new infrastructure to learn. More technologies to contend with that I have never gotten super good at like MySQL and MongoDB. The overwhelmingness and uncertainty of things like team makeup, on-call and such added to the technical things I had in front of me.

The rest of the week was a blur. I made a few changes here and there but in general avoided doing so. Even basic changes like setting up nscd seemed to cause problems. I really started questioning if I was in over my head.

This all got me thinking about my own fears and apprehension about things. First of all, I realized the emotions I go through with these kinds of new technical changes are standard for me. I tend to be overly cautious, hesitant to make changes, and easily spooked when the metrics don’t look right.

But why? Why do these things have such an affect on me? In many ways experience is to blame. Not just a lack of experience but the bad experiences along the way. I know what the command or action is “supposed” to do but I have come to question, is that what is actually going to happen? While I have faith in reading the docs, you just don’t know until you have run the command a few times what to expect.

While I understand security groups, etc. there is always the question of environment separation. I know that as long as I do the right things, nothing bad is going to happen and the world is going to be separate. That said my API keys have just as much power to kill prod instances as they do dev instances. And, as I have already discussed, I have a hard time trusting the tools.

Finally, the known unknowns kill me. The more I learn about MySQL the more I understand how little I actually know. While I will learn and become more familiar with its inner workings, I continue to be a bit hesitant to make large changes because, almost without fail, the database is the thing that breaks everything.

So what now? How do I work through these inner demons? I guess the simplest answer is to experiment and move forward. To remind myself that this is no different than any of the other times I have wadded knee deep into new infrastructure. To setup mock production environments and get a feel for what is actually going on. To do some reading and reach out to my amazing friends for help and advice.

That said, taking the weekend to step away and reflect has helped tremendously. So here is to an amazing new week ahead of me.

PuppetConf 2013

PuppetConf 2013 was an amazing experience. I could spend tons of time talking about all the things that went on but here are a few of the things that really stood out to me.

Developer Days and Open Spaces

The developer day stuff was great. While I didn’t get a chance to sit down and hack on much myself, it was great to be able to catch up with people in the community and talk about the things that are coming down the pipeline and need to be worked on.

The real win of Wednesday was the open space sessions that Michael Stanke put on. The discussions around testing and best practices/patterns were amazing. I feel like I got a ton out of just sitting and talking to others in the community. With that said, I am going to officially put it out there that I bought puppetbestpractices.com and my goal is to start collecting talks, slide decks and blog posts about patterns and module construction in one place. I haven’t decided how I am going to set it up yet, but I will make a decision soon.

Being a Speaker

This is probably the biggest speaking opportunity I have ever had. Puppet Labs not only did a great job putting on the conference but I felt super well taken care of as a speaker.

As I walked the halls I got a ton of positive response about my presentation. It was great to run into people and have them tell me, “great talk,” or ask questions about testing. I am looking forward to digging even deeper into testing and better understanding how rspec-system is going to become part of my workflow.

The Sessions

While I didn’t make it to as many sessions as I would have liked, the ones I did attend were great. I was impressed by how much knowledge each session was jam packed with. I can’t wait to go through and listen to all of the sessions as they are released.

In the meantime, the slides can be found at http://www.slideshare.net/PuppetLabs/puppetconf-at-a-glance.

The Hallway Track

Wow. I can’t even begin to list the tremendous number of amazing people I met while at PuppetConf. Whether it was getting a chance to catch up with @puppetmasterd himself or seeing old friends like @botchagalupe and @mitchellh the people are what really took the conference to the next level for me. Sitting down for, what must have been more than 2 hours, with @davemangot talking DevOps and systems architecture was one of the most enjoyable things I have done in a long time.

Lets just say, I am looking forward to PuppetConf 2014.

Taking Inventory

Every 6-9 months I find myself taking inventory of life in general. To be honest, this is usually a result of work getting busy for a sustained period of time. I find that it is important to take the time to go through this mental exercise as it is usually an indicator something needs to change about the way I am currently approaching life.

It is amazing how much this thought process actually reveals. Even the frequency in which I find myself walking through it gives hints about changes in life. Over the last few years, I have learned that if I am mentally taking inventory every month, it is probably time to change jobs.

This morning, as I started to take inventory on my morning walk, I realized two things about the process:

  • I want to be open about where things stand as an attempt to put myself at ease.
  • This is probably useful for others to see and at least think about.

So here are a few of the things that I ponder when taking inventory.

Family

Family is at the core of everything. As a husband and father of two, it is important to make sure I am taking the time for them and fulfilling their needs. The following are some of the things I think about.

Are my wife and I connected?

My wife has always talked about this idea of being connected. When we are connected, we anticipate each others needs, we are helping to carry the emotional load for each other and well, connected.

Am I spending enough time with my girls?

I don’t know that I will ever be able to answer yes to this question. I always find myself thinking I should be doing more. That is especially true during the summer. For my almost 4 year old, being outside and dance class are our thing. Mommy goes along sometimes, but, by and large, those are Daddy things. So when the normal high is 90+ degrees outside, the park becomes less of an option. To add to it, dance closes during the summer because many of the families go on vacation, etc.

This means the Fall is always one of my favorite times of year. Lots of trips to the park, dance is back in full swing and it is time for high school football. Because my wife is a high school teacher, we make a point to get out to most home football games and a few away games each year. It has always been fun to take my oldest out and I can’t wait to introduce my youngest.

Am I making time for my parents (and in-laws)?

My in-laws have always been super active in my married life with my mother-in-law watching my daughters during the day. We do spend a ton of time with them so things are usually pretty good. My parents on the other hand are always super busy. And not only are they super busy, their schedule is offset of mine by about 2-3 hours. They live in the same area but I am usually heading to bed as they are finishing with dinner. I feel like we are seeing them fairly often and with the normal Fall stuff coming, I think we will be good.

Work/Career

So work and career stuff is the other big area I think about. These questions are generally a bit more pointed and tend to help change my thoughts about long term things. In general, this area of my life seems to impact family and my happiness the most. It is also important to point out that there will be days when the answer to these questions is not positive, the key is to look at the general case over the last few weeks.

Am I having fun or do I enjoy going into work?

So simple, yet so powerful. When I get up in the morning do I look forward to the day ahead of me? Interestingly enough, this is almost a meta-question for me as it is really dictated by the other things I think about.

Is my relationship with my boss good?

I have been in a place where this was not true for a sustained period of time. It is no fun. It was during that time that I learned that if my regular answer to this is no, it should be a resume generating event.

That said, I love my boss. It has been amazing over the last eight months to have someone that asks hard questions and actively works to help me get over my own insecurities. It is awesome to work for someone who is honest and sees the bigger picture.

Is my relationship with my teammates good?

The team is what makes us stronger. It is the people you can lean on in times of distress and people to lift you up during times of good. Just like the relationship with your boss, your team makes a huge impact on overall happiness.

So, I’m going to be a broken record here, but I love my team. Such a great group. As the one remote guy, they make sure I don’t feel left out and they make the 3-6 hr round trip drive to Santa Monica each week worth it.

Is my work interesting?

I hate being bored. If my work is boring it means a few things:

  • I am probably not having fun
  • I am probably not learning much
  • I am probably not developing as an engineer the way I need to be

So decidedly, yes, my work is interesting. Right now it seems to be more a study of process and history but there are some neat technical challenges on my plate too. It is always nice to have a good balance.

Is my workload going to kill me?

It is no secret to my friends and family that I am a workaholic. While a great trait in some regards, it is something that takes its toll on occasion. The idea of only working 8-5 and not thinking about work outside of that time is crazy talk as far as I am concerned. I love the problem sets and the challenges so working a bit more is no big deal.

It is when I try to keep up with an impossible workload that I start to step back from work and stop enjoying it. Even worse, I do it to myself, taking on extra work that I don’t need to. My boss and one of my teammates have been amazing at helping me to back off and not let work consume my life. I am not sure that a lot of these questions would be answered positively if it were not for them helping me to keep things in check.

Am I doing enough career development?

As far as I am concerned, the technical aspects of career development, by and large come from work. It is important to make sure I am doing things at work that are in-demand and make sense for my career but as long as I am working in the right position that is generally not an issue.

The real place I think career development happens is at conferences, on this blog and, in many mays, on twitter. Am I submitting enough talks? Am I going to enough conferences? Am I blogging enough? These things matter because people matter. The social interactions and the friends in industry are really where it is at.

Now that I have had a chance to decompress and am feeling a bit better about the state of things, it is time to get back to work.

Why I Suck At Open Source

I love the idea behind open source and can’t imagine my career without the amazing contributions of so many people, but can’t help but feel that I don’t pull my own weight. I see amazing people in Ops like John Vincent (@lusis) and James Turnbull (@kartar) that are constantly shipping code and am reminded that I am not doing the same.

No matter how much I tell myself I need to push more code out into the world, I am regularly reminded that not only am I not great at writing good code, I don’t have the prerequisite knowledge about patterns and structure. To add insult to injury, the problems I find the most interesting are rarely general problems. The problems I find myself working on in my spare time are those internal issues that we can’t justify spending time on at the moment because there is so much else going on. So even if I am writing code, it either can’t be open sourced because it isn’t general enough, or I am too embarrassed to ask management to let me open source it.

When Packer was released it was a blast to have some time to go play with it. The bummer was that all I seemed able to do was report bugs and test to make sure they had been fixed. Every time I dug into the source code I found it difficult to make useful modifications to help solve problems. While I am usually able to read and help do basic code debug, I rarely find myself in a place where I have enough knowledge to make changes and contribute back.

While I know that evangelizing software and reporting bugs is useful and very much a part of open source, it is a bummer that I never seem to be able to get a Pull Request right.

Packer – First Thoughts

As I spent the weekend eagerly awaiting the arrival of my 2nd daughter and avoiding the heat, I got a chance to play with Packer quite a bit. It has been a blast getting it bootstrapped and spinning up VMs.

While there are still a few bugs to be worked out, packer seems very stable and well thought out. Everything from Ctrl-C doing what you would want by cleaning up after itself, to insightful error messages it is top notch software. A huge shout out goes to @patrickdebois whose veewee project made it trivial to get going with Packer. The veewee-to-packer gem, written by @mitchellh made it a snap to take my veewee definitions and turn them into Packer templates.

Why Packer?

For anyone that has been a long time user of Vagrant the first question that probably comes to mind is, “Why should I switch to Packer from veewee?” The simple answer is that Packer already does 90% of what veewee does, but it can build in parallel. Plus, Packer will handle the automagical building of VMware Vagrant boxes. In addition to those things, For me, the biggest reasons to use Packer are the lack of a need for a ruby environment, and the longer term extensibility that the foundation has already been laid for.

On the lack of a need for a ruby environment… Of all the things I have struggled with as an Ops guy is getting other members of my team to jump through the hoops that is getting a modern ruby environment setup. Between the version crap, bundler, etc. removing the need for ruby will be a huge win for me as I evangelize the use of Vagrant.

My Experience So Far

For being software that was at version 0.1.0 when I installed it, Packer is amazingly well documented. The community is already developing quickly and fixes for the bugs are rolling in super fast. I have hit a few problems here and there but almost all of them have already been addressed in version 0.1.3. I just want to say a huge, thank you, to @smerrill for all the work he done as it has made my life tremendously easier.

Getting Started

So then, where to start? If you are using veewee, I would suggest starting with the veewee-to-packer gem. If you haven’t played much with veewee and just want to start with Packer, @smerril has put up a great starter template for CentOS 6.4 at https://github.com/smerrill/packer-templates. My goal is to get some templates up in the next week or so depending on how things go.

The Future

Personally, I will be paying a lot of attention to Packer as things move forward. It make it a lot easier to duplicate what our corporate kickstart servers are doing, and will make creation of Vagrant boxes much easier and more accessible than it has been in the past.

Puppet Design Patterns

Programming and software engineering have never been areas where I would consider myself strong. Quite frankly, it is the exact opposite. Digging into code and troubleshooting is fine, but writing code from scratch is still something I struggle with. As I have jumped onto a team with a dev guy turned ops guy, discussions about how we write puppet code have come up.

Starting with a discussion about where ancillary services get instantiated (I’ll cover that in a separate blog post), I started looking for more details about design patterns in puppet. The more I looked and asked questions, the more I realized there is not much out there. At this point, the only real reference I have found was Simple Puppet Module Structure Redux by R.I. Pienaar. This article does a good job covering a self contained module that sets up a self contained service, but it really lacks some of the more advanced use cases.

My plan at this point is to start the conversation. Below is a list of issues that I think could be well served by design patterns and discussions of anti-patterns. As time goes by my goal is to start writing posts that help to better shed light on what I see as the issue and ways I understand that we can solve it. My hope is that this will start the discussion, because I am definitely not the one to suggest the right way of doing things.

Design Pattern Areas

  • Ancillary Service Configuration (Post is about 50% done)
  • Distinguishing between types of modules. (i.e. Type and Provider, Building Blocks, Roles, etc.)
  • Interaction with environments

Just incase it wasn’t clear, I am lost when it comes time to all of this stuff. My hope is that those that have been there and done that can help shed some light on the right ways to move forward.

Update

So the first response I got to this post in IRC was to take a look at a Designing Puppet – Roles and Profiles by Craig Dunn. This is a great starting point for a lot of the thoughts I have had.

Puppet On Windows: First Thoughts

One of the things I didn’t expect when changing jobs was the amount of Windows Infrastructure I would be responsible for. While not expected, it will be an interesting challenge and is still very much a WebOps and not an enterprise Windows deployment. As we move forward with this infrastructure, there is a huge desire to puppetize it and manage it in a programatic way.

Setting Up a Dev Environment

As a long time Vagrant user and abuser, I started with attempting to setup a base box using Veewee and Vagrant. This was met with disaster. None of the templates I tried worked and, really, vagrant isn’t designed to handle Windows. But failure of this kind always seems to open new doors.

So, I cheated a bit. The veewee template left behind a floppy image with all the of the autounattend.xml stuff rolled inside. I created a new VM in VirtualBox, added a floppy controller, the virtual floppy disk and the Windows 2008 R2 iso. A few minutes later, I had a fully installed Windows 2008 R2 instance.

From there, I shared a folder with my Puppet manifests and modules, installed Puppet and the VirtualBox Guest Additions. At this point, I took a snapshot so I can easily revert back to where I started.

Out with the Package

So, what is the first thing you want to do when setting up a web server? Install the web server software, right? On a Linux box this looks something like:

1
2
3
4
5
6
7
8

package {"httpd":
  ensure => installed,
}

service {"httpd":
  ensure  => running,
  require => Package['httpd'],
}

Simple enough, right? So I set out to figure out how to install IIS. The first thing I looked for was a built-in type to handle setting up server roles and features. Nada. So off to the Forge I went. I found an amazing IIS module written by Simon Dean. The only problem with it, it doesn’t install IIS.

Enter DISM

DISM or Deployment Image Servicing and Management is where the heavy lifting gets done. is the command line tool that allows you to install roles and features, just like you would from the Server Manager GUI. I owe a huge thanks to Mark Bools for helping shine the light on DISM with a blog post. To get started with dism in puppet, Puppet Labs has published the dism module on the forge that has a type and provider to work with. To get a list of things that can be installed with dism you can run, . From there, you can use the dism type to install the needed packages. This looks something like:

1
2
3
4
5
6
7
8

dism { 'IIS-WebServerRole':
  ensure => present,
}

dism { 'IIS-WebServer':
  ensure  => present,
  require => Dism['IIS-WebServerRole'],
}

General Observations

In general, things worked as expected. Puppet does not seem to care whether the manifests have Windows or UNIX line endings which is very nice. The only thing I have found so far that didn’t, “just work,” was the module tool, otherwise things translate just fine. The big issues left to deal with are not ones that are easily solved by puppet. There are reboots that need to take place as part of basic installations that really don’t lend themselves well to configuration management as a whole.