DevOps At the Crossroads: CMDB and the "Drift" Problem

Ok, I admit it. There's still a bit of the mischievous adolescent still in me.

Today, I threw a paradigm "stink bomb" into a lunch conversation between an ITIL guy and a DevOps (very Agile minded) practitioner by casually asking: "Is there a place for Configuration Management System/Configuration Management DataBase (CMS/CMDB) in DevOp cloud automation?"

As soon as that topic puck hit the table, the verbal hockey game began in earnest. I've changed the names, not to protect the innocent, but to protect myself from reprisals.

DevOps and CMS/CMDB

DevOps: "There's no place for a formal CMDB in DevOps, that's backward thinking. Creating a centralized, system configuration information repository, managed outside the development team's build-test-deploy pipelines just adds another unnecessary layer to slow down moving development and operations closer together so they sit the same boat."

ITIL: "If there's anything I detest more than moldy blue cheese, it's when I hear you Agile guys tearing down any attempt to put the structure of any kind about the SDLC and IT infrastructure. It's always a fight to have catalogs of assets and how things are put together because you guys think it's all virtual and a CMDB or CMS is only for physical data centers."

DevOps: "CMDB is just another silo and an outdated way to make 'brick-and-mortar' build-test-prod environment rigid and inflexible, fighting rapid change to keep pace with technology. CMDB is like building a pharaoh's tomb with all the IT infrastructure buried in it for the afterlife."

ITIL: "I suppose GIT source control doesn't store artifacts, and history of the software changes and development process effort huh? If CMDB entombs IT infrastructure the way you claim, and you as so against that, then why use version control to write code changes huh? Git is the Agile's "hieroglyphics" of tracking software changes. And what about those jenkinfiles, dockerfiles, you git repo huh? Are you really telling me that's not building a configuration database?"

... And round and round and round this went for almost forty minutes .. with no decisive knock out, no clear winner. But it was entertaining to watch.

Configuration, Snowflakes, and the Drift Problem

You haven't been in DevOps very long if you haven't heard a developer howl, "But it worked (ran, built) just fine on my (laptop, desktop, etc.)" at least a hundred times by now. Aggravating as this still is to an Ops Engineer, or Prod engineer the release/ "QA" manager, what adds insult to the injury is it seems to never dawn on the development folks the infrastructure engineer can't fix a problem describe by the "fuzzy" end of things.

You haven't been in IT cloud or on-prem data center engineering long if you haven't heard about the "Drift Problem." Keeping DevOp system components in sync and up-to-day takes continuous effort and monitoring to ensure the "gap" between Git commit and prod is as small as possible. DevOps engineering work is never done, and mastering the waves and cycles to balance release velocity and developer productivity takes more the ingenuity and technical skill for its' often like trying to dance between the raindrops in a downpour to avoid getting wet.

Combating Drift, for example, means not letting your current CI/CD pipelines and automation, however well designed, become monuments - DevOps engineers should not be technology archeologists. A little creative destruction is a good thing from time to time and there is technical debt in all code, even infrastructure, and automation code. Avoiding technology traps like not moving from virtual machines to containers, and from containers to clusters, from clusters to serverless service delivery, etc. Architecture is a buzz word to many DevOps engineers, and if it's is to you, then come down a notch and think about 'blueprinting'. Microsoft Visio can get your CI/CD automation engineering cut down to manageable size, especially if you put the time into becoming familiar with SysML using the online tutorial by Incose (International Council on Systems Engineering).

DevOps vendors are pushing so many shiny new automation toys into the market these days, it's like Christmas. Think "lean" from day one and avoid getting caught up in the tulip bulb mania their slick ad campaigns create. Engineering is a pragmatic profession where it's critical to know your tools and how to use them. Practical knowledge of a few key CI/CD automation tools is more valuable than "brochure deep" familiarity with an arsenal of DevOps tools. Also, don't create any snowflakes as you progress through your engineering your DevOps CI/CD automation. Build, test, in small steps again and again. A little Chaos Engineering can go a long way in locating weak spots and if Netflix can loose the monkey during production, maybe you should also.

Configuration Management By Any Other Name

DevOps complexity is not easily tamed; and like reality, when you ignore it then it automatically works against you. Combating a swarm of one-off dev environment configurations which are almost impossible to troubleshoot, synchronize, maintain, and standardize and the hailstorm of requests they can create won't happen on its own. If you haven't set up a DevOps SLA, then give doing so serious thought as a way to bring sanity to how you meet DevOps service demands. Service Level Agreements are not a silver bullet but they can give you leverage when you have to cost- and performance justify your CI/CD automation decisions. And taming the complexity definitely won't happen without a baseline, plan, and collaboration with key stakeholders. DevOps is a non-stop service operation with almost no time for meetings and ceremony. Scrum is a good model for all sorts of stakeholder focused development. You can do a fifteen-minute stand-up with your development team to plan automation sprints, get a backlog going -- especially if you're the sole and solitary DevOps engineer.

Data-Driven DevOps: Know Your Numbers, Use Them.

Nowadays, almost every CI/CD system available generates tons of data. Unfortunately, few DevOps engineers take the time to sift through whats being dumped onto their screens every day. Set aside time to track, analyze, and act on your key performance indicators. If you don't know your KPIs, or haven't been capturing them you'd better start right away. Use the data your systems give you, but avoid the decision-making trap of "availability bias" and link what you measure and monitor to results that matter so your DevOps planning and execution activities are not just reactive firefighting.

DevOps engineering has to be a numbers game: to win, you have to play. This is the key takeaway. If you don't track the right metrics then whatever you try to change or improve will be a 'fire-ready-aim' exercise. DevOps engineers aren't hired to be statisticians or Excel wizards. Luckily tools like RStudio and Anaconda can ease your burdens and give you an easy way to crunch your numbers and graph them to see trends. Getting your hands dirty in your DevOps automation data can bring your engineering from gut-level artistry to a science and point you in the right direction to reach a new level of DevOp system performance whereas demand grows and things have to scale, handing the additional demands will get easier as they do.

Data-driven DevOps is not a mainstream idea, but it can allow you to make the best use of your limited time and resources as you try different approaches to balancing cost and performance of your cloud-based CI/CD infrastructure with a better chance to measure whether they are effective or not. In time, you'll know where, when, and how to invest effort and energy in your DevOps automation strategy. Also don't forget to create a feedback loop to capture data from the developers, testers, and release teams so you know you are moving in the right direction.

Over time, data can allow you to build strategies to justify a staged, capability-based, continuous improvement effort and to drive your CI/CD metrics in the right direction to increase quality, reduced delivery time, and increase productivity. Regardless of how well your DevOps automation performs, it's the increased developer satisfaction and dev-team performance that's the top priority. Your best developers will not tolerate a CI/CD pipeline system that does provide them with the right support, tools, and a build-test-release environment that supports them.

DevOps Configuration Management: To CMS or Not to CMS, that is the question

Doing DevOps by the seat of your pants may work for a while. But eventually, your CI/CD will become fragile and nothing more than a house of cards. You can choose a CMDB or another 'repo' system to keep track of the myriad of pipeline configuration details: the goal is to have a "blueprint" to build your build-test-deploy infrastructure. DevOps popularity may keep your company's CTO from looking too closely at your budget, but that won't last. Eventually, you'll have to prove your CI/CD infrastructure, and all of its cloud-based components, is earning its keep and its' growth (both CAPEX and OPEX) is driven by KPI you have monitored and can use to forecast what's ahead.