To Err is Human; To Automate, Devine?

I can usually tell how a day will go before I even get out of bed in the morning. How? Pick up my smartphone and peruse my inbox. My rule of thumb is that the quantity of emails time-stamped before 8am is indirectly proportional to quality of my day. I know I’m not alone here.

On most days the typical Network and Systems Administrator has to deal with 10 – 12 dashboards displaying status, hundreds of alerts from their monitoring tools, and read/generate/react to a multitude of reports. Even for the optimist who assumes most days are good that doesn’t sound like a great scenario to me. Add in a bunch of failures and angry end-users and you’ve got yourself a recipe for a “Yuck” sandwich.

To carry our food metaphor a little further let’s call “Automation” our key to a better tasting day. What things should the typical administrator consider for automation? Some tasks are urgent and some are important. However, before we dive right in and suggest HAL-9000 or Skynet to take over your NMS let’s clarify our definitions.

Urgent tasks are things that need to be done right now. Do not pass “Go”. Do not collect $200. Drop what you’re doing and fix the problem. You want to automate urgent tasks so that you don’t get caught in that reactionary, fire-fighting mode. We’ve all found ourselves in this situation. It’s inefficient and unsustainable. Urgent tasks are a huge time-suck and rarely contribute to your long-term plans.

In contrast, important tasks are rarely urgent but always contribute to your long-term plans. When you plan important tasks ahead of time, they add value to your team and to your organization’s mission.

Stephen Covey, known for his book “7 Habits of Highly Effective People”, suggested a mechanism he called the Eisenhower Decision Matrix. It was homage to a quote attributed to the former President: “I have two kinds of problems, the urgent and the important. The urgent are not important, and the important are never urgent.” Note below the graphical representation of this matrix and how it translates to the day-to-day activities of typical Network and Systems Administrators.

Given those definitions and the above graphic our goal is to automate as many of those things as possible starting in quadrant 1 and working our way around. The more tasks we can automate the more efficient, more effective, and less prone to human error our operation becomes.

Quadrant 1: Important and Urgent

What are things that typically fall into the “Important and Urgent” category for network and systems administrators? Chief among them is alert handling. When your tools generate hundreds of alerts, it’s impossible to respond to all of them. Coupled with a two prong approach of reducing alert fatigue seek out a toolset that can automate alert responses. The same is true for root cause analysis and infrastructure management. Find tools that assist you in your day-to-day activities. 

Quadrant 2: Important, Not Urgent

Of course, there are lots of aspects of everyday IT administration that are important, but not of “drop everything” urgency. The best example here is report generation. We all know those “Pointy-Haired” bosses that either can’t/won’t create IT reports used for decision support. At same time, those of us on the front lines don’t have time because we’re dealing tasks discussed in Quadrant 1. This seems like a prime candidate for an automatable task. The same could be said of Infrastructure visibility. Wouldn’t it be great to simply click a button and have a complete view of your infrastructure in the palm of your hand?  

Quadrant 3: Not Important, Urgent

Moving into the next category of things to deal with we consider those tasks that are urgent, but not necessarily important to address. Keep in mind in this instance “Not Important” doesn’t mean we don’t care about it, but they’re items that we can do without. Translation: Make your tools do it for you. The consequences of cloud and SDN technologies playing a bigger part of every organization’s IT infrastructure is that once-static architectures become more dynamic and fluid. You want to make sure your NMS tools automatically discover these resources and the system can care and feed for itself 

Quadrant 4: Not Urgent, Not Important

Last, but not least, are the things that don’t need to be done right away and aren’t particularly important. Unlike the Covey version matrices, with regard to systems and network administration, these aren’t trivial or time-wasting tasks. Instead they’re things we’d throw into the classification of “If it ain’t broke, don’t fix it”. For example, if your NMS system is functioning as needed (i.e. no security problems or lack of compelling features), then why engage in the “make-work” project of upgrading it? If your NMS tool has a built-in and flexible reporting engine why spend time integrating it to your BI tools? These are things that don’t need to be automated.

Now that we’ve got our lingo straight, what to automate (plus their priorities), and why to automate, the question of “How?” must be answered. In the final analysis there are three strategies in moving forward:

A) BYOS – Bring Your Own Scripts. Roll your own solution to add automation to your infrastructure

B) COTS – Commercial Off The Shelf. Find a ready-made product that meets your needs.

C) BOBW – Best of Both Worlds. This approach is hybrid. Look for a commercial product that is feature-rich, while at the same time is solution-oriented and capable of accommodating custom-written code where required.

Imagine a world where every morning the sun rises, the birds tweet a welcoming tune, and your inbox isn’t filled with oodles of alarms that may or may not be signaling the collapse of your IT infrastructure. That scenario doesn’t have to be a fantasy. Making an effort to automate your network and systems management can get you there … well minus the singing birds and sunrises.

Those are things that are out of any vendor’s control.

About the Author: Andy Markowitz

Andy has 20+ years of experience in the software industry. In that time he's been a sysadmin, project manager, sorta' developer, and done extensive work in both post-sales support, pre-sales solutions engineering. Andy holds a B.S in Computer Information Systems from Colorado State University and an MBA from Cal State Fullerton. "There is no better career field than Information Systems. You get the best of both worlds: the geekiness of Computer Science and the human interaction of the business world. I feel like I won the Lottery (almost) everyday with what I've chosen to do for a living." When he's not thinking about how to help customers solve their IT visibility challenges he's concentrating on the truly important stuff in life: Being a fantastic husband and trying to keep up with his two energetic small children.