DevOps at the Crossroads: Failure is an Option

Failure sucks but instructs. — Bob Sutton, Stanford Management Instructor

Unlike prior posts, this one discusses why DevOps initiatives fail. DevOps vendors seldom talk about failure, given the historic and persistent rates of IT project failures, topping 60% in most project categories for the past two decades, there’s no reason to conclude DevOps projects are immune.

While there is no shortage of CI/CD technology and other bright, shiny automation tools, these are only the visible ‘above the waterline’ part of successful DevOps implementation. The key success factors in creating a productive and sustainable DevOps are below the waterline: orchestrating organization and cultural change. DevOp technology vendors haven’t offered tools to deal with this part of the DevOps iceberg. 

Failure is Always An Option

Discouraging as project success rates are; optimism, confidence, and belief in DevOp success is still high. Straddling the fault line intersection where IT operations and software development collide, the tectonic shift to reduce silos and unify the thinking of both groups on achieving DevOps is luckily something few DevOps practitioners have had to do. Fewer still have had to face a conference room filled with senior managers, many of who are not only dubious of the DevOps idea but who have a sizable investment in the status quo separating IT operations and software development. 

But quieting doubt and disbelief are nothing compared to facing a room brimming with antagonistic and disappointed executive decision-makers, and convincing that group to stay the DevOps course after repeated failures or lack of measurable progress and benefits. The saying, “..it’s like learning to play the violin in public” captures the essence of such moments. Evangelizing DevOps against their opposing voices that say, “It’s just another tech bubble.” or “We’re not big enough to need that here.” verge on the heroic, even the quixotic.  

DevOps and The Art and Science of Innovation

DevOps is fundamental organizational change and a disruptive paradigm culture shift. Like so many other disruptive innovations, the field is an elephant graveyard where many forward-thinking; imaginative system administrators or development project managers have lost their career bones. DevOps has several steep learning curves and climbing them involves enduring the ‘off-key’ moments while slowly and painfully learning to play the music correctly while time, money, and patience run out.

The steepest DevOps learning curve is charting the course for organizational change and overcoming entrenched cultural mindset separating IT operations and software development. DevOps initiative leaders strive to achieve incremental change that eventually accumulates into breakthroughs. Despite having or building DevOps competencies in engineering and IT data center operations, success requires understanding how to evolve how people think and behave and what motivates them.  

Organizational change is recognized as the most demanding and risky efforts undertaken by innovators and evangelists. There’s seldom a clear, well-marked path to follow to master the art and science of creating something new or changing what already exists. Yet there is a process and a discipline of successful organizational change; one that follows an uncertain and torturous path to create practical solutions and bring them to fruition. 

As the diagram shows, organizational change involves transitioning people (roles, responsibilities), processes (internal, external, and shared), structure (team organization, positions, reporting relationships) as well as technology across a divide between a current system and a future one.

Included in organizational change transition, is the understanding and recognition of how the change will affect the relationships between those current and future organizational elements which are described above as 'dimensional contingency' It's the misunderstood, misinterpreted, or ignored differences between the technology's dimensional contingency and that of the organization which sows the seeds of failure.

Why DevOps Fails: Lessons Learned

The surveys of unsuccessful DevOps initiative have created a poster child for DevOps failure: an organization where IT operations and the development groups first embrace the idea, but then fail to commit to making it a part of their shared organizational DNA. DevOps evangelists, whether they come from either the IT operations or software development silos, ride the rush of initial enthusiasm, but eventually find themselves unable to overcome unexpected cross-currents and white waters of resistance from both sides along with waning upper management commitment as the organizational challenges and culture conflict problems mount.

There are at least, a few early warning signs of DevOps failure and the lessons learned emerging from project post-mortems and surveys do follow a surprising, but not unexpected pattern. A general summary of why DevOp initiatives fail is found when noticing its’ similarities to why IT projects in less developed countries fail. 

DevOps Organizational Change: Contingency Theory

What does change look like in an IT organization? The British researcher, Richard Heeks studied why successful IT projects unexpectedly failed when tried in the developing countries where the people and culture were different. Heek’s explained the reasons IT projects fail using an organization change model like this one. 

Heeks’ claimed most IT projects transferred to developing countries failed because of an unrecognized “gap”, not because of poor technology match or lack of know-how and resources. Heeks also explained how this gap caused failures that weren’t just black or white but occurred by degrees that made it difficult to define IT project success and failure. Many IT projects he studied succeeded in various ways when their goals, objectives, and methodology were modified to align more with the culture of the organization and their ability to change. But measuring that success was not done based on the originator's goals, milestones priorities, and objectives for the technologies.

Crossing DevOps Chasm: the "Design-Actuality" Gap

Heeks explored IT project failures and came up with what he called the “design and actuality gap” failure model” He discovered most IT projects, like DevOps, fail because at the outset they don’t recognize how implementing a technology, even a proven successful one, requires an environment of organizational and culture changes. These changes are not immediately apparent or visible in the technology itself and are often ignored or dismissed – primarily because technology creators can’t provide the tools to address them.   

DevOps Design-Actuality Gaps

DevOps evangelists who either had to start from scratch of build on prior efforts to bring IT operations and software development together may not have understood the challenge of closing the design-actuality gap. Caught up in building the CI/CD operational infrastructure, providing the training and tools to use it, whether in their on-premise data center or in the cloud, DevOps evangelists have encountered ‘gap’ problems without recognizing them or why they occurred. 

Since the spectrum of organizations and cultures, even of firms with similar characteristics such as size, age, and market focus, within a specific industry or a group, is diverse, the majority of the DevOp initiatives haven’t considered the similarities between the challenges they experienced bridging their silo gaps and the gaps challenges of transferring IT technology between developed and developing countries. Yet Heek’s insights about IT project failure echo several comments and lessons learned examples DevOps evangelists have offered to highlight the risks and challenges that have to be addressed and overcome.

DevOps: Closing the Gap

Heeks summarized the differences in the organizational change components that contribute to failure categorized by the "hard" versus "soft" elements in his design-actuality gap model as follows:

DevOp Failure Take-Aways

The essential takeaways from Heeks work on the hard and soft elements in his design-actuality gap mode applied to DevOps failure could be summarized as follows:

DevOps and most information technologies claim to be based just on “hard”, tangible skills and competencies, but they also depend critically on “soft” skills and capabilities such as creating change across domains of people and making technology adaptations for cultural changes.

·     DevOp evangelists and stakeholders base their plans on the premise "If it (the technology) works for others, then it'll work for us too" mentality that ignores differences between the “as-is” and “to-be” worlds DevOps seeks to create between IT operations and developers. Implementing DevOp technology does not “automate” the change to bridge those gaps.

·     DevOps technology ignores differences and gaps between the organizational and cultural environment needed to unify both IT operations and development and assume what’s needed for success either already exists or will be created by the technology alone.

·     DevOps failure, whether total or partial, occurs frequently because of unaddressed, unrecognized gaps between “hard” design (technology) and “soft” actuality (organizational and cultural environment).

·     DevOps success is never simply, black or white, success or failure, but occurs steadily by degrees, iteratively, by accepting the uncertainty and delays in such progress which is often not acknowledged or handled properly. 

Heeks, R. “Information systems and developing countries: failure, success and local improvisations”, The Information Society, 18(2), 101112, 2002