Observability, Monitoring, WTF

One of the more interesting observations I made early in my career was that if you asked engineers a question you would rarely get a response… but if you made a statement that was clearly wrong, they would take all the time needed to explain to you why you are wrong.

Duty Calls
Credit: xkcd

So with that in mind, below is my attempt to reason about monitoring, observability, alerting, etc. I am going to reason about it in terms of the tools because without a practical way to reason about an implementation it just feels like a philosophical debate. I am sure I am wrong about much of it and am looking to understand why.

Concepts

Telemetry

This is the data being emitted from the system, it comes from all over the place and isn’t necessarily being used by anything, but it is there. In the best case it has lots of metadata, but much of the data will just be a number. The interesting thing here is that most of the useful telemetry, IMHO, from the platform is flowing into tools that many of us in operations and engineering don’t even have access to.

Monitoring

I like to think about monitoring as the gauge cluster on my car. It tells me incredibly important things; that I can usually get by without. Let’s be honest, many of us that have been driving for a while could get by without almost all of the gauges on our car. Even the fuel gauge is redundant because we would learn to estimate how far we could go between fill ups and as a backup carry an extra gallon or two in the car.

black-car-instrument-cluster-panel-945443
Image Source

The interesting part is that we would never (for most values of never), do this. You would never want to operate a vehicle without some basic gauges and this is true of our infrastructure. We want to have basic understandings of things like CPU, Disk, HTTP errors, etc. But here is the crucial thing… those bits are only interesting when you understand the context around them. Sure, the engine overheating is always bad, but is the speed of the car being 70 MPH always good? Even on the same stretch of road? Even worse, as many of us have learned, all the gauges can look fine and your car can still be in a bad spot.

Monitoring for me is the set of signals I choose to watch for. It is the things I “know” what good and bad looks like in 95% of cases. As the system gets more complex there are more gauges, and getting poked to go look at them isn’t going to help anything. In the modern world, the value is going to be in finding as few things for humans to actively watch as possible with the rest of the cases just being conditions in the system that are handled automatically.

Observability

This is when things go wrong or we are tuning. Continuing with the terrible car analogy, we only want to look at the high cardinality detailed events of what is going on in the car when we are tuning it to go faster or when it is in a bad state and we are trying to figure out which part of the car to go take apart. The battery gauge says zero charge, but is the issue the battery? The alternator? The wiring harness?

The best part of this example is just how bad this is in a car and how many of our systems match this problem. If I am racing Formula 1 or NASCAR they are going to be able to look at the telemetry from the car and know exactly what the problem is. And, by the way, it is being sent wirelessly in real time. For those of us that are not at companies like Facebook or Google, we basically have to do things like swap in a new battery or charge the battery eternally to figure out the problem. The idea with observability is being able to get back to the source code of the problem as quickly as possible. What is actually causing the issue? The thing that companies like Honeycomb and projects like OpenTelemetry seem to be wanting to bring us is the ability to get the Formula 1 tech available to us in our Kia.

NASCAR
Image Source

What resonates the most with me is that much of the observability bits will not be things that generate alerts. It will be user reports and monitors that are wonky that will cause us to dig out our observability tools and figure out what is wrong. Don’t get me wrong, the observability tools will emit metrics, but that will not be the core focus.

Classes of Tools

Metrics

These are the raw numbers that the system is generating. Load, CPU, disk consumption, requests, etc. This is by far one of the most mature parts of the stack at this point. On a positive note, SNMP based metrics collection has mostly gone away at this point. The tools I associate with this space:

  • Datadog
  • Statsd
  • Prometheus
  • SignalFX
  • collectd
  • telegraf

Interestingly enough, the storage and visualization pieces are in many ways separate from the actual metrics gathering bits. In the visualization space, Grafana is the current modern winner with Graphite being the previous generation. On the storage side, I don’t know that I have heard mention of anything other than InfluxDB in a long time.

reparing-791413_1920

With the tools figured out, the core thought here is that the metrics gathering mechanisms almost entirely map to the monitoring side. These are the systems that power the gauge cluster. I still think it is important to have raw metrics, but in the debugging of a system, they frequently send you on wild goose chases. Much like the gauges, I can solve simple problems with my metrics, like the disk being full, but in most cases it doesn’t actually get me closer to solving the underlying problem.

Tracing

This is the actual instrumentation of code and tracing it through execution. When I look at this space Datadog has an offering, but Honeycomb seems to be the only vendor that is actually digging in deep on this side of the world. In the OSS space, the tools are basically OpenTelemetry (which is a merger of the OpenCensus and OpenTracing projects), Jaeger, and Zipkin.

To me, this is the heart of the observability side of the world. The traces are what allow you to understand what is going on in the system. Much like that little you got from running NewRelic on PHP systems, it is the ability to to visually see where things are going wrong and dig in. This is the type of diagnostic data the race car drivers are using. It is what gives those big systems companies a leg up when dealing with their systems.

The most important thing about this is the tying of the events that are happening in the system back to the code base. It should not be hard to go from a span in Jaeger or Honeycomb to the source code that generated the span. It is also trivial to tag your spans inside of the system and you get timing for free.

Listening to Charity Majors and Liz Fong-Jones talk about this and having played with tracing a bit, I am convinced that we should be moving away from a LOT of the logging we do and instead be shoving most of that data into spans that are reported back to something like Honeycomb or Jaeger.

Logging

Logging is the bane of my existence. The reality is that the way we do logging today, it really isn’t useful in anything more than an Auditing context. The massive effort to index and collate the logs so you, just maybe, will know the right incantation to put into Splunk is killer. As I said above, I am moving away from the log everything world to the create a span for everything world.

Synthetics, APM, and Checks

I mention these here because they are such a broad set of tools that do ALL THE THINGS. Some of the synthetics platforms are insanely great observability tools, helping you to dig into a waterfall of the page load, giving you clear understandings of what is actually going wrong. The problem is that most of these tools are rediculously expensive and their own ecosystem. I really want a day when I can pass a token from a Catchpoint synthetic test into my system and be able to link it up to a set of traces in the rest of my system.

There are other very obtuse signals that come from these systems. Pingdom and Nagios style checks are their own thing. They are binary, up or down. These are monitors, but only monitor things of last resort. Is it up? That question is much less important in the modern app stack. Don’t get me wrong, you should ask it, but if that is the only thing you are looking for, a 200 at the front door, you have bigger issues. That is like asking if the car started but not looking to see if it will even shift into gear or move.

Other tools

There are a few other classes of tools that we should be discussing in terms of observability and monitoring that frequently get left off the list.

From a monitoring and telemetry perspective there is HUGE amount of information being gathered by marketing and product systems. Google Analytics, Segment, Adobe Audience Manager, etc. Have you ever wondered when people use the site? How about what they do when they are on the site? What browsers are used most frequently? Product knows.

On the other side of the coin there are the exception tracking systems like Rollbar or Sentry. These systems are incredibly useful in observing how the code is breaking in production, assuming you are using a language that throws exceptions, etc.

TL;DR

Monitoring and observability are both needed and serve different purposes. You need monitoring to handle the knowns and observability for the unknowns. Yes, it is still important to know about cases where you are regularly running low on disk, but it is the observability practice that will let you figure out what stupid thing the code is doing to cause that.

Configuring Envoy to Use SSL/TLS with the v2 API

I have been doing a bit of playing with the Envoy Proxy this week. One of the things I ran into that has been painful was configuring a listener to use SSL/TLS. Below is some sample config to make it easier for the next person to dig out the config necessary to make it happen.

static_resources:
  listeners:
  - address:
      socket_address:
        address: 0.0.0.0
        port_value: 443
    filter_chains:
      tls_context:
        common_tls_context:
          tls_certificates:
          - certificate_chain:
              filename: "/etc/envoy/frontend-certs/service.crt"
            private_key:
              filename: "/etc/envoy/frontend-certs/service.key"
      filters:
      - name: envoy.http_connection_manager
        config:
          tracing:
            operation_name: egress
          access_log:
          - name: envoy.file_access_log
            config:
              path: "/dev/stdout"
          codec_type: auto
          stat_prefix: ingress_http
          route_config:
          (…)

Software Leadership

It is interesting how even slight changes in the way things are working cause you to re-evaluate your path. When I first started as the manager of my team, the focus really was around, well, learning to be a manager. Even more than that, the nature of the work was very small piecemeal projects, or projects that were very stand alone. This worked to my advantage, as it gave me a chance to focus on being a manager.

This concept is interesting in and of itself. With a great mentor to help me drive in the right direction, I was able to understand and move forward in that direction. Interestingly enough, one of the largest things I got was a chance to learn was how to balance the need to jump in as an IC on occasions to get the job done.

A huge part of this journey was driven by reading and book learnings. As much as you learn on the job, there is a dramatic need to read and understand the state of the world. Anyone that has talked with me about my journey in the last few years will have likely gotten a reading list to go with the conversation. I think this is a super important thing. Reading and keeping up with the concepts that are being revealed is super huge. It is also not enough to just skim the books, you really do have to read them, or in my case, listen to them.

My thoughts as a people manager have been shaped heavily by the following books (in no particular order):

In the last few years, I have read/listened to many more including books like Radical Candor and Good to Great averaging at least one book a month, usually more.

This habit has really helped me to learn a ton, but even more than that, it has helped me to see patterns in my life. Recently, it became very apparent that there was a shift in the work that needed to be done. First, the business books were really starting to repeat themselves. Each new book explained the same basic concepts in slightly different ways. The a-ha moments got further and further apart. Then, very rapidly, my book list changed dramatically.

So the journey continues, but it is now much more heavily focused on software leadership and architecture instead of general management. While both are super heavily intertwined, I really do see distinct differences. So here comes the change…

In case you are wondering, the next books that ended up on the revised book list:

And the list goes on…

Manager vs. Individual Contributor

One of the more common things I have heard over my time as a manager is the idea that you can be an Individual Contributor or a Manager… Pick one. They are different jobs, etc, etc, etc. As we hear in the blog posts, as an Engineering Manager I shouldn’t be writing code, I shouldn’t be solving the problem. Instead, I should be working on leading my team, and focusing on their career growth and helping them to find their path.

Like so many other things, I ended up at this false dichotomy, a choice of either, or. For me personally, I just accepted that I wasn’t mature enough in my journey and would figure out how to not get involved in the technical details like writing code at some point. Some day I would grow up and would be enlightened enough to understand how to do that or I would have enough staff so I didn’t have to write code or manage instances. The problem is, as I realized tonight, that I will fail as a manager if I stop doing those things.

The reality boils down to really understanding my role as a manager. It is my job to lead and grow my team of individual contributors towards our shared vision. Understanding this job really lies at the heart of what we do. It is my job to understand how my team grows, learns, and even more so the trajectory they are on. Writing code or doing “Individual Contributor” work is not a bad thing, in fact it is a good thing.

My time working at Carls Jr. in high school really made all of this clear. Think of the General Manager at a fast food restaurant. If she sits in the back and tells folks what to do, but doesn’t jump on the line while there is a rush, something is horribly wrong. During large rushes, I have seen regional directors, VPs, and general managers jump in to help. What is particularly interesting is that unlike many of the other members of the team, they know how all the things work. While they may not be as good as the best cook, they surely can hold their own and manage to get a Famous Star or a Spicy Chicken out to a customer. And if the thing that needs doing is helping people to prioritize how to get orders in bags or on trays they can do that do. The point is, it isn’t their job to be the best at any one position, but they have to be able to jump in and fill that role or at least know who they can move to solve one problem while they address another.

So why does this matter as an engineering manager? My job is to grow and lead the team toward that shared vision. Sometimes that is going to mean setting an example, sometimes it is going to mean letting folks find their own way, and other times it may mean doing the work yourself because it doesn’t make sense to have others do it. The key to all of this comes back to leading and growing. When I find myself doing individual contributor work I have to ask, how is this contributing to the team? Am I robbing someone of a chance to grow in a way they really need to? Or the flip of that… By doing this work myself, is it going to enable me to give the team greater context around the state of the world and more autonomy later? By doing this work, am I allowing people to instead focus on things that matter to their career path?

With that realization in mind, I can already feel the weight of doing individual contributor work leaving my shoulders. Maybe I am not doing the wrong thing as much as I thought…

Why I don’t blog much anymore…

Writing about tech was always time consuming and not always easy to get into the right mind frame to do. But at least when I did it, it was concrete and I could expose my assumptions. I believe using Habitat to do can be explained, and I can even wrap the argument in assumptions. Docker this, sensu that, a little orchestration to solve this problem.

Contrast that with management. Am I a good manager? I think so? I hope so? My boss would have said something if I was sucking, right? Are the things I am doing just side effects or are they actually doing what I expect? There really is no way to be sure. This makes it incredibly hard to write about… I can tell you the books I have read, I can even on occasion, explain why I feel the way I do. But. to put down concretely that I am doing it right seems so artificial and likely a lie.

The things I know for sure:

  • I have an awesome team.
  • I have an awesome boss.
  • There is more experience to gain, no matter how many books I read.

So the question becomes… How do I share? How do we share? How do we grow together as managers the same way we have grown together as technologists? Going a step further, how do we prepare each other for the types of things we are going to encounter?

Databases, Pipelines, and Failures

One of the more interesting things that comes up for me when building new pipelines is around handling the moving of already running services and how things play out when disaster recovery is in play. The hardest part has been framing the question. What are we trying to achieve and how does that affect the other things we are trying to achive? My starting point has been around this:

  • What is the “right” behavior in normal operations?
  • What is the “right” behavior in disaster recovery or new pipeline build out?

All of this really gets interesting when discussing a service that is heavily database dependent, like WordPress. In an ideal world, each deploy into acceptance, union, and rehearsal would restore either a replica of exactly what is in delivered/production right now or what was there fairly recently (think last night’s backup). This is actually super easy to achive. Assuming that the data we need is accessible, we can build that into the pipeline.

Except, what happens when the production system isn’t there for reference? When acceptance goes to be deployed before we get to delivered how do we get valid data in the database? I really see two options:

  • Fail the pipeline and force manual intervention.
  • Initialize the database with an empty set of data.

When we deploy most applications, at this point, the second option is quite normal. When you are building out for the first time it makes a tremendous amount of sense. Really, this question probably doesn’t even cross your mind because there is no production data to start with. On the flip side, when you look at WordPress and many enterprise applications, much of the config of the application is actually stored in the database. Not to mention, when moving an already existing application, your concerns are usually a bit different than when you are building a new application.

The benefit of forcing the fail is that it ensures that you don’t accidentally deploy without a working application. Whether it be settings or content, WordPress without a restore of the data isn’t very useful if it has been used much already. The problem is that, this is hugely disruptive. The first time you spin the pipeline all the way through, you end up needing to fix four database installs (acceptance, union, rehearsal, delivered).

The place I have landed is asking a few more questions:

  • What is the overall importance of this application?
  • Does this fit into the dependency chain in a way that it is going to block other applications if it doesn’t deploy?
  • What impact does this have on the system as a whole?
  • Rephrasing the above question, if we have bad data, are other systems going to do the wrong thing?
  • If other systems do the wrong thing, what could the impact be?

The place that this brings me to is really around the nature of the service and the larger implications. Today I am moving around a WordPress instance. That instance happens to support the blog and our events page. By it being up with wrong data durring a disaster recovery situation, it really isn’t going to hurt anything. It is important, but to have a few pages 404 for a day or two is not the end of the world. Turning that around, it is in the dependency chain for the main website. Which means, in order for us to get the site deployed the app needs to work. There is much more stress created if we cant deploy www because of a missing wordpress backup.

The hypothetical enterprise system also helps justify this position. If instead of a WordPress instance we were talking about an internal payroll system, it may actually make a ton of sense to fail the pipeline each time. We really do want to ensure that data is correct because other systems do act on it. In a time of crisis you dont want your benefits system to send all of your employees notification that they are now eligible for COBRA and that their health coverage ends in a few days.

SysAdvent Stats

November is a hectic time of year for me and a few years back I made it even crazier by volunteering to take the reigns of SysAdvent, the annual advent calendar for SysAdmins, Ops, DevOps, and all the other folks that are excited about systems. As I have been working to arrange some sponsorship stuff I got asked a super reasonable question about stats. So instead of replying in email I thought I would share the lifetime stats from the blogger console.

Stats > Overview

Overview

Stats > Posts

Posts

Stats > Traffic Sources

Traffic Sources

Stats > Audience

Audience

Of all the data presented here, it is crazy to see that Day 1 – Docker in Production: Reality, Not Hype, an article from 2014 is in the top 3 articles and it is less than a year old. It is super exciting to see the growth of SysAdvent and exciting to see how many folks visit the site each day.

I can’t wait for this year’s SysAdvent.

Delivering My New Reality

Back in late December/early January, I took on a new role as the Engineering Lead of Corporate Infrastructure and Applications at Chef. As I started to understand my new role, one of the things I set out to do was figure out what our standard stack and deploy patterns were going to be. As fate would have it, in early March I got a ping in from one of the folks in marketing about a crazy idea they had about the ChefConf keynote.

With three and a half weeks left before ChefConf, he said, “we want to deliver the announcement about Delivery on stage at ChefConf using Delivery.” Within about two weeks I had built a new pipeline, using Delivery, that would get our corporate website, https://www.chef.io, deployed. As an aside, the decision to move to Fastly as part of this was probably almost as important as moving to Delivery itself.

For those that missed it… Seth Falcon and James Casey actually did deliver https://www.chef.io to the world that day using Delivery, live and on stage. Never in my life has a few minutes felt so long.

I am now a convert. The idea of managing an environment operationally, without Delivery or something very close to it, makes me shudder. It has fundamentally changed the way I think about what I do as a professional. The key distinguishing fact, for me anyway, is that we use Delivery to deliver services, not just cookbooks, not just application code, but services.

At this point, my team is already delivering lots of things with Delivery:

The crazy part is that there are so many reasons why I am in love with Delivery that I can’t actually explain them in one post. But my plan is to start writing about each of them. The following is a list of things I plan on writing posts about:

  • The standard way of shipping things: Making changes is the same, no matter what app you are in.
  • The artifact is king: Shipping a promotable artifact really is a HUGE win.
  • The shape of the pipeline: Where the phases and stages are makes a huge difference in how I think now.
  • Yes, you really do need four separate environments.
  • A down and dirty intro to build cookbooks.
  • It can’t be all rainbows and unicorns… A look at some of the pain points and gotchas.

I really am super excited about Delivery and not just because I work at Chef. Hopefully even if you aren’t using Delivery itself, some of the things I have learned about CD will help you as well.

Stubbing encrypted_data_bag_for_environment

We use the encrypted_data_bag_for_environment method from the chef-sugar library pretty heavily in the cookbooks my team uses. With that said, I can never seem to remember the right invocation of stubs and mocks to be able to test recipes that have those calls in them. Since I don’t want to have to come up with this again, here is an example of how I did it this morning. In this example, I am stubbing a data bag item that has creds for using DataDog.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

require 'spec_helper'

describe 'cia_infra::base' do
  let(:chef_run) do
    stub_command('which sudo').and_return('/usr/bin/sudo')
    Chef::Config['encrypted_data_bag_secret'] = '/dev/null'
    @runner = ChefSpec::ServerRunner.new
    @runner.converge(described_recipe)
  end

  before do
    allow(Chef::EncryptedDataBagItem).to receive(:load).with('cia-creds', 'datadog', '').and_return({
      'default' => {
        'api_key' => 'datadog_api_key',
        'application_key' => 'datadog_application_key',
      }
    })
  end

  it { expect(chef_run).to include_recipe('apt') }
  it { expect(chef_run).to include_recipe('chef-client') }
  it { expect(chef_run).to include_recipe('chef-client::delete_validation') }
  it { expect(chef_run).to include_recipe('ntp') }
  it { expect(chef_run).to include_recipe('push-jobs') }
  it { expect(chef_run).to include_recipe('oc-users') }
  it { expect(chef_run).to include_recipe('cia_infra::monitoring') }
end

It is entirely possible that I am doing this wrong, but hey, it works. I would love to hear better approaches if there are any.

Adventures in PTO

For anyone that was watching Twitter last week, you probably noticed that all of Chef Software headed to Seattle for Rally Week. You may have also noticed that I wasn’t there. I was on PTO. It is a culture that allows this sort of thing that has really given me confidence in the “Unlimited” PTO model and really has me thinking about all the negatives folks have pointed out.

Is it even PTO?

At a very fundamental level, it really seems as if the reason there is even a need for unlimited PTO policies is because we have forgotten what it means to be salaried employees. I have worked the spectrum… At one place of employment, if you worked at all that day, it was technically a work day and could not be considered PTO. On the flip side, I have worked for employers where it was hard to “leave early” at 4:00 pm (even after starting my day close to 6:00 am) and if I took off before 3:00 pm it was expected that I put a half day off in the time tracking system. The fact that it wasn’t out of the normal to do 10 and 12 hour days had no impact on whether I “took PTO.” Contrast those with Chef where it is tracked to the point that I put an entry on the calendar if I take the entire day off, otherwise, it is between my manager and me.

Growing up, the way I understood being a salaried employee meant that most weeks you would work more than 40 hrs but it also wouldn’t be ridiculous to work 30 hrs in a week, because you were expected to get your job done rather than to punch the time clock. Right down to the point, my dad used to point out the lawsuits where folks that were salaried tracked their time and were judged against it and won tons of back pay. Being salary meant that I needed to show up when it made sense for my job and leave when it made sense as well. If I took a long lunch, who cares?

The point being, if I take a day to go volunteer, but am still available to answer questions is it really PTO? I think in many ways the unlimited PTO policies are meant more to address this more than anything. If you need to go do some volunteer work to help make you happier, go do that. If taking the morning off to get your oil changed and run a few errands allows you to focus in the afternoon, you should probably go do that too. If you have just had a morning that sucked and you really just want to go surf to take your mind off of work, why are you still in the office?

Vacation

So if taking a day, a morning or an afternoon here and there isn’t PTO, what is? For me, it is vacations. For some that means stay-cations for others it means travel. Whatever it is, it is a chance to disconnect and not worry about being available at all. (I don’t tend to disconnect fully, but that is a discussion for another time.) It is this sort of PTO that I think most folks struggle with.

Travis CI, a company based in Germany, recently discussed a “minimum” PTO policy where they discussed requiring folks to take a minimum of five weeks. To put that in perspective, I have never had more than three weeks a year, and at one job there was compulsory vacation that had to be taken during the holidays. (Yes, you could go without pay, but you were expected to take PTO.) Additionally, there are usually caps on how much vacation you can accrue, usually around 300 hrs. Notice the other word in that statement, accrue. You don’t just start with that thee weeks, you have to earn and accrue it. So if after you have been with the company a month you want to take a week off for a family vacation, you are out of luck. What I am getting at is that we as Americans, tend to not be very good at taking vacation.

We struggle to take this time off because there is alway so much to be done. One of the best lessons I ever learned was that the work wasn’t going away. If I took off a week, the work would still be there for me when I got back. And if that specific work needed to be done before I left, there was always work that could wait. That isn’t to say I don’t work hard and try to get done lots and lots, it is just a realization that the work will fill the time allotted, so it is up to me to limit that time. Much of what makes unlimited PTO hard is that because you didn’t “earn” it, you don’t always see it as yours to take. Frankly, if you are working your hardest, you have earned it.

One of the smartest, stupid decisions I ever made was buying into a timeshare. While you can argue the financial outcome of owning timeshare, what I will argue is that the impact on my mental health has been well worth any financial benefit or loss involved. The reason it has been beneficial is that it forces the issue. I take vacation because I have already paid for it. As much as there is something relaxing about hacking on stuff while sitting on the balcony overlooking Los Archos in Cabo San Lucas, I feel guilty that I am not taking advantage of the amazing place I am visiting.

Managers make all the difference

Ok, so company culture plays a huge role too, but managers really are at the heart of making unlimited PTO work. The best way to explain this is actually an experience recently… A few months back I went to Disneyland in the afternoon after my daughter was done with school to catch up with my parents and my cousin who was in town from Wisconsin. When my mom asked about work, I replied, “well, I knew that if my boss caught wind that I stayed at work and gave up this rare chance to catch up with family, I would catch shit, so I got in the car and came.” At Chef, I have regularly been encouraged to take the time I need. When the question of going to the all hands week vs going on vacation with family during a week we travel almost every year came up, I honestly had a harder time with the idea of me missing it than my boss did. At the end of the day, not only has my manager, but all of my coworkers as well, reminded me that family comes first.

As I take on a management role, I will strive to build that same sort of culture into my team. PTO is important. Family is important. Above all, being happy is important. So if that means taking a day here and there to go volunteer, or go surf, take it. If what makes you happy is taking off for a few hours on Tuesday and Thursday afternoons to go teach a class, how can I support you in doing that? Additionally, that means setting a good example. I really do see it as my job as a manager to take those days on occasion and to go on vacation. In fact, my next vacation is already planned, is yours?