I have been doing a bit of playing with the Envoy Proxy this week. One of the things I ran into that has been painful was configuring a listener to use SSL/TLS. Below is some sample config to make it easier for the next person to dig out the config necessary to make it happen.

static_resources:
  listeners:
  - address:
      socket_address:
        address: 0.0.0.0
        port_value: 443
    filter_chains:
      tls_context:
        common_tls_context:
          tls_certificates:
          - certificate_chain:
              filename: "/etc/envoy/frontend-certs/service.crt"
            private_key:
              filename: "/etc/envoy/frontend-certs/service.key"
      filters:
      - name: envoy.http_connection_manager
        config:
          tracing:
            operation_name: egress
          access_log:
          - name: envoy.file_access_log
            config:
              path: "/dev/stdout"
          codec_type: auto
          stat_prefix: ingress_http
          route_config:
          (…)

It is interesting how even slight changes in the way things are working cause you to re-evaluate your path. When I first started as the manager of my team, the focus really was around, well, learning to be a manager. Even more than that, the nature of the work was very small piecemeal projects, or projects that were very stand alone. This worked to my advantage, as it gave me a chance to focus on being a manager.

This concept is interesting in and of itself. With a great mentor to help me drive in the right direction, I was able to understand and move forward in that direction. Interestingly enough, one of the largest things I got was a chance to learn was how to balance the need to jump in as an IC on occasions to get the job done.

A huge part of this journey was driven by reading and book learnings. As much as you learn on the job, there is a dramatic need to read and understand the state of the world. Anyone that has talked with me about my journey in the last few years will have likely gotten a reading list to go with the conversation. I think this is a super important thing. Reading and keeping up with the concepts that are being revealed is super huge. It is also not enough to just skim the books, you really do have to read them, or in my case, listen to them.

My thoughts as a people manager have been shaped heavily by the following books (in no particular order):

• First, Break All The Rules
• Leaders Eat Last by Simon Sinek
• It’s Your Ship by Capt. D. Michael Abrashoff, USN, Ret.
• Team of Teams by Gen. Stanley McChrystal, USA, Ret.
• Extreme Ownership by Jocko Willink and Leif Babin
• The Toyota Way by Dr. Jeffrey Liker
• Silos, Politics, and Turf Wars by Patrick Lencioni
• Start with Why by Simon Sinek
• The Goal by Eliyahu M. Goldratt

In the last few years, I have read/listened to many more including books like Radical Candor and Good to Great averaging at least one book a month, usually more.

This habit has really helped me to learn a ton, but even more than that, it has helped me to see patterns in my life. Recently, it became very apparent that there was a shift in the work that needed to be done. First, the business books were really starting to repeat themselves. Each new book explained the same basic concepts in slightly different ways. The a-ha moments got further and further apart. Then, very rapidly, my book list changed dramatically.

So the journey continues, but it is now much more heavily focused on software leadership and architecture instead of general management. While both are super heavily intertwined, I really do see distinct differences. So here comes the change…

In case you are wondering, the next books that ended up on the revised book list:

• Domain-Driven Design by Eric Evans
• Implementing Domain-Driven Design by Vaughn Vernon
• UML Distilled by Martin Fowler

And the list goes on…

Writing about tech was always time consuming and not always easy to get into the right mind frame to do. But at least when I did it, it was concrete and I could expose my assumptions. I believe using Habitat to do can be explained, and I can even wrap the argument in assumptions. Docker this, sensu that, a little orchestration to solve this problem.

Contrast that with management. Am I a good manager? I think so? I hope so? My boss would have said something if I was sucking, right? Are the things I am doing just side effects or are they actually doing what I expect? There really is no way to be sure. This makes it incredibly hard to write about… I can tell you the books I have read, I can even on occasion, explain why I feel the way I do. But. to put down concretely that I am doing it right seems so artificial and likely a lie.

The things I know for sure:

• I have an awesome team.
• I have an awesome boss.
• There is more experience to gain, no matter how many books I read.

So the question becomes… How do I share? How do we share? How do we grow together as managers the same way we have grown together as technologists? Going a step further, how do we prepare each other for the types of things we are going to encounter?

The last week at PuppetConf was an absolute blast. It was great to catch up with all the amazing folks doing amazing work. But, there was one thing that bothered me quite a bit. In many different contexts I heard people more or less say, “I can’t write Ruby, I’m not a coder.” Or, just as bad, “I use Puppet because I am not a coder.”

These sentiments bother me in so many ways that I could probably sit and rant for half the day. But really, after thinking about it quite a bit, there are a few key reasons why this idea just doesn’t sit well with me. I really think that if you are someone that says something similar to the above, you should rethink things.

The Technical

Puppet is why I know Ruby

I feel like an old man in saying this, but back in my day, we didn’t have this fancy facter.d stuff. We had to write our custom facts using Ruby and we liked it. Ok, so maybe the liking it part is a bit of a stretch, but I definitely liked the results. When I wrote those facts, I didn’t have any clue what a method was or how objects worked, I just knew that when I pasted the right things in and tweaked them a bit, I was able to get information I needed, and that was awesome.

I don’t actually know Ruby

So, anyone that actually thinks I know Ruby, probably has never actually looked at my code. While I am not scared to fire up irb or pry and copy pasta some code in to solve problems, the idea of a large scale Rails or Sinatra App does not sit well with me. I am absolutely a coder, but I am by no means much more than a Junior Ruby Developer when it comes time to my Ruby skills. Like most good SysAdmins, I just happen to be good at Google.

Have I ever mentioned how much I dislike CPAN?

Whenever I went to go do anything in Perl, I found myself reaching for a module from CPAN. The CPAN topic is probably worth its own discussion, and, to be fair, has probably gotten a bit better since the last time I tackled this. But, as a result of the pain of CPAN and the nature of the environment I was working in at the time, installing new Perl modules wasn’t an option and even if it was, it wasn’t really feasible.

Enter Ruby. As a result of running Puppet everywhere, I had Ruby everywhere for free. Combine that with the fact that the Ruby Standard Library is kinda awesome, you get the ability to do some phenomenal things. Since most of my job involved ALL of the systems, I needed to be able to write scripts that would work everywhere in our infrastructure. With the power of Ruby’s Standard Library and the fact that the code was actually readable, even a week later in most cases, it made a great way to do systems stuff, everywhere.

Level up your Puppet game

The real power in Puppet can be found in writing custom functions and custom types and providers. It is the custom functions that let you actually move some of the crazy that usually gets done inside of templates, thus hidden from plain view, and move it back into the manifest where it can be seen in context with all the other things going on in the manifest. The custom types and providers give you the benefit of being able to manage ALL THE THINGS in Puppet. Do you want to manage DNS about systems inside of Puppet? How about adding systems to that new fancy monitoring SaaS? All of that gets done in custom types and providers.

The Squishy Stuff

You are already a coder

Are you writing Puppet Code? Are you using ERB? If so, you are already a coder. You are already dealing with making decisions about the APIs you present or don’t present. You are dealing with control structures and code organization. You are a coder.

Code is eating the world

So I am not going to actually dig into this except to say, code is becoming an integral part of our lives as Operations and Systems Administration Professionals. Whether it be with Chef, Puppet, Ansible or the next thing, being a coder is going to be at the center of the jobs that we do.

After an amazing adventure with the great people at Demand Media, I on my way to cook up something new. If the cheesy pun didn’t give it away, I will be joining the community team at Chef on April 28th. While I will miss the amazing folks at Demand, I am super excited about the new adventures ahead.

My role at Chef will be working as a Community Software Engineer working to help make the experience of using and contributing to community cookbooks delightful. I am looking forward to working with all the amazing people in our community.

I am looking forward to the new adventures with Chef and the ability to focus on making the lives of others better as I work with people to make it easier to solve problems with cookbooks. The ability to focus on shared and open-source infrastructure code is something I have wanted to do for a long time. This will give me the chance to help encapsulate the years of yak shaving into code that will hopefully let others do more awesome.

Just because I want to explicitly state it, I still have lots of love for Puppet and the folks at Puppet Labs. I am excited to continue to see where the Puppet ecosystem goes. While I for sure have some definite reasons why I like Chef, Puppet is still an amazing tool.

If you are going to be at #ChefConf I will be around all day on Wednesday so hit me up.

In early December of last year, I hired on at Demand Media to be an 80% remote employee. While, in general, the culture at Demand around being in the office is strong, there are actually a fair number of remote employees at Demand. For me personally, I live about 90 miles from our office in Santa Monica and usually come in on Tuesdays.

The journey has definitely had its ups and downs but it has been amazingly positive and enjoyable. When I first considered the idea of working for Demand, I reached out to a few friends I knew that were also remote. The interesting thing about this is that they all advised against me taking the position because not only was it partially remote, but because I was going to be “the remote guy” on a team that was in the office. After having spent a year in that world, I can definitely see why they had concerns and how quickly those situations could turn sour.

Because I am a little on the crazy side, I took the job. It was a chance to get some experience working remote and the team seemed like a good fit. Needless to say, I made the right decision. I feel like I have been super successful working at Demand and owe a huge part of that to the team I work with. While tools help a lot, it is the culture of my team and the other teams I work with that allow me to be effective.

My team, the Media Infrastructure group, takes an active role in making sure I am not left out of things. They always make sure the conference calls are setup and I am there via video chat when possible. Not only that, there is an active effort to make sure conversations are had over IM so that I can be a part of them and am not left out. On the occasions where I am not able to be a part of whatever was going on in the office, they make a concerted effort to follow up with me and make sure I am aware of what is going on.

In addition, the teams I interact with have adapted to the fact that I am remote and do a great job keeping me in the loop. There are plenty of screenshots of the teams being on a video chat with me as we launched a new site, or you can walk into one of the weekly lunch meetings to see my gigantic head on the video display. Most of the developers I work with have made a point to make sure they have Jabber up so I can reach out to them when there are issues.

The impact of tools in this space cannot be understated. The conference bridges, Jabber and the Vidyo system are what allow all of this to be possible. I want to take a second to thank the amazing teams in my IT group that made the Vidyo system happen and keep things like Jabber up and running. While tools like Google Hangouts and other instant messaging systems exist, the tools that IT provide work better and have allowed for a much more seamless environment. In particular, when we were in a previous office location, I used to dread meetings where I had to call in on the conference phone. Half the time I couldn’t hear what was being said and the other half the time I couldn’t chime in when I needed to. The Vidyo systems that are now in the big conference rooms have made that problem go away almost completely, and I am now one of the loudest people in the room.

It is interesting to look at some of the benefits of being remote. Most people focus on what the remote employee gets out of it, but I think the employer gets the most benefit, especially with the industry we are in. As a result of being remote, my team regularly practices the same techniques we use during incident response. We all know how to get on the conference bridges and are comfortable working and IMing at the same time. (Ok so that is a bit of a stretch… I suck at multi-tasking.) Even more interesting, when I get a call in the middle of the night about a problem, I generally walk into my normal work environment so there is less of a need to adapt to the new situation. I turn on the display I have setup as an information radiator that displays dashboards and logs and am able to quickly start getting a feel for what is wrong, even if I still have to setup my laptop.

Another interesting thing that has happened on a few occasions is that because I am in a different location, I am not affected by local issues. There was a situation recently where the power had failed in the office and because I was remote and one of my teammates was still at home we were able to address issues that couldn’t be handled by the rest of our team in the office. It means that I am occasionally able to route around problems or see things differently because I am not in the office.

I would be doing a disservice if I didn’t talk a little about the challenges that I have faced over the last year. Most of the problems I have faced are not ones that can be overcome by my team or the company, they are the reason that remote work isn’t for everyone.

To start, 90 miles is a long way. Without stopping and without traffic I can do the drive in approximately 1hr and 20 mins. For those that have ever been to LA, the “without traffic” part is, by and large, hard to achieve. To help with that I have adjusted my schedule a bit and try to be out of the house by 4:00 am on days I am in the office, resulting in about 1hr, 30 minute drive to the office. And, because traffic in the evening is bad, I don’t head for home any earlier than 6:00 pm and it is usually closer to 7:00 pm by the time I leave resulting in about a 2hr drive home. While I do come in twice a week on a very rare occasion, it is obvious by Friday that I have done so by the toll it takes on my body.

To add to the theme of long days, I tend to spend a lot of time working. This is partially because I just love what I do so much and partially because I really don’t ever leave work. I try to be good about getting up from the computer no later than 6:00 pm, but that doesn’t always happen and it isn’t out of the ordinary for me to sit down to dinner with the family and then be right back at it. My wife had been amazingly understanding and has done a great job in helping me to set those boundaries and not get sucked too far into the work I enjoy so much.

While my team does a great job at keeping me included, there are still plenty of things I miss out on because I am remote. I don’t get to participate in a lot of the things like the Monthly Birthday and Anniversary goodies most months, and I am almost never in the office for lunch on Friday.

Additionally, there have been a few occasions here and there where there has been a major incident or something going on and lots of brainstorming and discussion was going on in the office and I sat at my computer wondering what was happening because everyone had stopped responding. This has more or less gone away because of Vidyo, but it is a horrible feeling when you are sitting at your computer, the site is down or heavily impacted and you feel helpless, not knowing what to do next.

All and all, it has been an amazing experience. My team, the teams I interact with and everyone at Demand has always been supportive. While the drive is a bit crazy and the day is long, I love my one day a week in the office and am looking forward to the next few years at Demand.

As I start to get things in order for puppetbestpractices.com, I am stuck on a relatively big decision, how do I setup the site? I basically see this two options:

1. I setup WordPress and grab a pretty theme and some plugins.
2. I take a bit more time and setup the site using a static page generator like Jekyll.

The first aspect that makes the decision hard is collaboration. If I use Jekyll or some other static site generator, I can throw the page up on GitHub and anyone that wants to contribute can just create a pull request. This makes contributions super simple. WordPress has the opposite problem. The only way I can accept contributions is to get emails and comments, or to hand out accounts. Neither of of those options are appealing.

From there the question of design comes into play. I almost titled this post “Design Eye for the SysAdmin Guy” because I suck so hard at it. IMHO, WordPress is the clear winner in this battle. It has tons of free and easy to use themes etc. Jekyll or maybe even Octopress have a much more limited selection of themes and by and large require me to actually know what I am doing, which, as I mentioned before is not the case. If the themes from something like Themeforest made sense from a license standpoint, I would gladly pay the roughly $15 bucks for a starting point.

So yeah, this has been my frustration since PuppetConf, how to build the site. I guess this is just a reminder that anything worth doing is going to be hard.

On Friday of PuppetConf I jumped on a video call with my coworker back at the office to get news that the president of engineering had asked for me to join the team supporting the new site that we had recently acquired. Everything about this was super exciting. It brought with it the chance to finally do something production in AWS and the potential to be a little more focused on a single property.

While I have generally kept up with how AWS works and have dabbled a bit here and there, I have never done anything that would constitute production. I had heard about the occasional shit show that is EBS and was able to grok how all the components came together. In my excitement, I spent the weekend using Cloud Formation to spin up simple HA setups and then tying in New Relic for monitoring.

Then Monday hit. Like a ton of bricks. So many unanswered questions and so much new infrastructure to learn. More technologies to contend with that I have never gotten super good at like MySQL and MongoDB. The overwhelmingness and uncertainty of things like team makeup, on-call and such added to the technical things I had in front of me.

The rest of the week was a blur. I made a few changes here and there but in general avoided doing so. Even basic changes like setting up nscd seemed to cause problems. I really started questioning if I was in over my head.

This all got me thinking about my own fears and apprehension about things. First of all, I realized the emotions I go through with these kinds of new technical changes are standard for me. I tend to be overly cautious, hesitant to make changes, and easily spooked when the metrics don’t look right.

But why? Why do these things have such an affect on me? In many ways experience is to blame. Not just a lack of experience but the bad experiences along the way. I know what the command or action is “supposed” to do but I have come to question, is that what is actually going to happen? While I have faith in reading the docs, you just don’t know until you have run the command a few times what to expect.

While I understand security groups, etc. there is always the question of environment separation. I know that as long as I do the right things, nothing bad is going to happen and the world is going to be separate. That said my API keys have just as much power to kill prod instances as they do dev instances. And, as I have already discussed, I have a hard time trusting the tools.

Finally, the known unknowns kill me. The more I learn about MySQL the more I understand how little I actually know. While I will learn and become more familiar with its inner workings, I continue to be a bit hesitant to make large changes because, almost without fail, the database is the thing that breaks everything.

So what now? How do I work through these inner demons? I guess the simplest answer is to experiment and move forward. To remind myself that this is no different than any of the other times I have wadded knee deep into new infrastructure. To setup mock production environments and get a feel for what is actually going on. To do some reading and reach out to my amazing friends for help and advice.

That said, taking the weekend to step away and reflect has helped tremendously. So here is to an amazing new week ahead of me.

Every 6-9 months I find myself taking inventory of life in general. To be honest, this is usually a result of work getting busy for a sustained period of time. I find that it is important to take the time to go through this mental exercise as it is usually an indicator something needs to change about the way I am currently approaching life.

It is amazing how much this thought process actually reveals. Even the frequency in which I find myself walking through it gives hints about changes in life. Over the last few years, I have learned that if I am mentally taking inventory every month, it is probably time to change jobs.

This morning, as I started to take inventory on my morning walk, I realized two things about the process:

• I want to be open about where things stand as an attempt to put myself at ease.
• This is probably useful for others to see and at least think about.

So here are a few of the things that I ponder when taking inventory.

Family

Family is at the core of everything. As a husband and father of two, it is important to make sure I am taking the time for them and fulfilling their needs. The following are some of the things I think about.

Are my wife and I connected?

My wife has always talked about this idea of being connected. When we are connected, we anticipate each others needs, we are helping to carry the emotional load for each other and well, connected.

Am I spending enough time with my girls?

I don’t know that I will ever be able to answer yes to this question. I always find myself thinking I should be doing more. That is especially true during the summer. For my almost 4 year old, being outside and dance class are our thing. Mommy goes along sometimes, but, by and large, those are Daddy things. So when the normal high is 90+ degrees outside, the park becomes less of an option. To add to it, dance closes during the summer because many of the families go on vacation, etc.

This means the Fall is always one of my favorite times of year. Lots of trips to the park, dance is back in full swing and it is time for high school football. Because my wife is a high school teacher, we make a point to get out to most home football games and a few away games each year. It has always been fun to take my oldest out and I can’t wait to introduce my youngest.

Am I making time for my parents (and in-laws)?

My in-laws have always been super active in my married life with my mother-in-law watching my daughters during the day. We do spend a ton of time with them so things are usually pretty good. My parents on the other hand are always super busy. And not only are they super busy, their schedule is offset of mine by about 2-3 hours. They live in the same area but I am usually heading to bed as they are finishing with dinner. I feel like we are seeing them fairly often and with the normal Fall stuff coming, I think we will be good.

Work/Career

So work and career stuff is the other big area I think about. These questions are generally a bit more pointed and tend to help change my thoughts about long term things. In general, this area of my life seems to impact family and my happiness the most. It is also important to point out that there will be days when the answer to these questions is not positive, the key is to look at the general case over the last few weeks.

Am I having fun or do I enjoy going into work?

So simple, yet so powerful. When I get up in the morning do I look forward to the day ahead of me? Interestingly enough, this is almost a meta-question for me as it is really dictated by the other things I think about.

Is my relationship with my boss good?

I have been in a place where this was not true for a sustained period of time. It is no fun. It was during that time that I learned that if my regular answer to this is no, it should be a resume generating event.

That said, I love my boss. It has been amazing over the last eight months to have someone that asks hard questions and actively works to help me get over my own insecurities. It is awesome to work for someone who is honest and sees the bigger picture.

Is my relationship with my teammates good?

The team is what makes us stronger. It is the people you can lean on in times of distress and people to lift you up during times of good. Just like the relationship with your boss, your team makes a huge impact on overall happiness.

So, I’m going to be a broken record here, but I love my team. Such a great group. As the one remote guy, they make sure I don’t feel left out and they make the 3-6 hr round trip drive to Santa Monica each week worth it.

Is my work interesting?

I hate being bored. If my work is boring it means a few things:

• I am probably not having fun
• I am probably not learning much
• I am probably not developing as an engineer the way I need to be

So decidedly, yes, my work is interesting. Right now it seems to be more a study of process and history but there are some neat technical challenges on my plate too. It is always nice to have a good balance.

Is my workload going to kill me?

It is no secret to my friends and family that I am a workaholic. While a great trait in some regards, it is something that takes its toll on occasion. The idea of only working 8-5 and not thinking about work outside of that time is crazy talk as far as I am concerned. I love the problem sets and the challenges so working a bit more is no big deal.

It is when I try to keep up with an impossible workload that I start to step back from work and stop enjoying it. Even worse, I do it to myself, taking on extra work that I don’t need to. My boss and one of my teammates have been amazing at helping me to back off and not let work consume my life. I am not sure that a lot of these questions would be answered positively if it were not for them helping me to keep things in check.

Am I doing enough career development?

As far as I am concerned, the technical aspects of career development, by and large come from work. It is important to make sure I am doing things at work that are in-demand and make sense for my career but as long as I am working in the right position that is generally not an issue.

The real place I think career development happens is at conferences, on this blog and, in many mays, on twitter. Am I submitting enough talks? Am I going to enough conferences? Am I blogging enough? These things matter because people matter. The social interactions and the friends in industry are really where it is at.

Now that I have had a chance to decompress and am feeling a bit better about the state of things, it is time to get back to work.

I love the idea behind open source and can’t imagine my career without the amazing contributions of so many people, but can’t help but feel that I don’t pull my own weight. I see amazing people in Ops like John Vincent (@lusis) and James Turnbull (@kartar) that are constantly shipping code and am reminded that I am not doing the same.

No matter how much I tell myself I need to push more code out into the world, I am regularly reminded that not only am I not great at writing good code, I don’t have the prerequisite knowledge about patterns and structure. To add insult to injury, the problems I find the most interesting are rarely general problems. The problems I find myself working on in my spare time are those internal issues that we can’t justify spending time on at the moment because there is so much else going on. So even if I am writing code, it either can’t be open sourced because it isn’t general enough, or I am too embarrassed to ask management to let me open source it.

When Packer was released it was a blast to have some time to go play with it. The bummer was that all I seemed able to do was report bugs and test to make sure they had been fixed. Every time I dug into the source code I found it difficult to make useful modifications to help solve problems. While I am usually able to read and help do basic code debug, I rarely find myself in a place where I have enough knowledge to make changes and contribute back.

While I know that evangelizing software and reporting bugs is useful and very much a part of open source, it is a bummer that I never seem to be able to get a Pull Request right.