Can Public Cloud save the CMDB?

As a veteran of the IT Service Management (ITSM) industry, I have been involved in many failed Configuration Management Database (CMDB) projects, and am aware of many more. In the early days of the service management movement, organizations attempted to implement CMDBs as a central component of IT Infrastructure Library (ITIL) best practices. The CMDB promised visibility into the upstream and downstream dependencies of applications and infrastructure, so that IT organizations could more easily plan for change (e.g., “we are planning a change, what are the potential upstream and downstream impacts?”) and more effectively respond to outages (e.g., “we have an outage….what changed upstream and downstream that could be the root cause?”). The CMDB promised to make IT organizations more effective and efficient while delivering a higher level of service. It all made perfect sense on paper, but proved immensely more difficult to implement in practice.*    

Example of an business service and application dependency tree stored in the CMDB 

The Challenge

The fate and failure of many CMDB projects can be traced to several common dynamics: 

1. Lack of Mandate: The decision to purchase and implement a CMDB was often undertaken without the involvement or commitment from the application teams. Since application releases would always be more important than a CMDB effort (and since all software releases are always and everywhere are some flavor of “late”), project managers would struggle to get the right stakeholders with meaningful application knowledge to participate in planning & discovery meetings, if they could get them to attend at all.
2. Lack of Perceived Value: In a traditional and siloed IT organization (ironically the ones most in need of the promise of the CMDB), application teams would tend to only see value in a tool or system that provided more information about their respective applications than they currently had at their disposal. Since the CMDB was about upstream and downstream impacts, the value was often lost on application owners, especially given the amount of work and subsequent care and feeding the CMDB typically required. Application teams would be much more likely to fund an Application Performance Management (APM) tool such as AppDynamics or Dynatrace than invest in the CMDB experience (see #1 above).
3. Complicated Discovery Tools: In order to build service, application, and infrastructure dependency mapping in the CMDB, one of two discovery methods is typically required: a.) agentless discovery or b.) agent-based discovery. While agentless discovery is more lightweight and requires less red tape and deployment overhead, it requires the discovery engine to be granted elevated access privileges within the domain, which leads to negotiations with network, security, and server level personnel (again, stakeholders sometimes not involved in the CMDB mandate) to provide access. In a poorly documented or poorly understood legacy IT infrastructure, this almost surely would lead to stoppages as new levels of infrastructure were discovered, and the negotiations would begin again for access rights. While deployed agents are a better way to map application dependencies, this too can be a battle as there are almost always “too many” agents already running on any server, and adding “just one more” tends to meet resistance unless the application team sees value (see #2 above). Even after securing agreement from the application teams, there is typically a struggle to get it deployed, as it is “never a good time” and schedules/priorities can conflict between the application/server and service management teams. Both of these challenges above induce an inevitable drag on timelines and burn on budget. 

The common results from the challenges outlined above would often lead to reduced scope for discovery and implementation, limited to only the most important application portfolio, the easiest application portfolio, or any application whose owners had bought into the project. While this would allow the project team to declare victory, it would result in an incomplete picture within the CMDB. This tendency to redefine victory would often prevent the CMDB from serving as a trusted source, and human nature dictates that the first time a CMDB is queried and the information is missing or incomplete, the CMDB becomes branded as “useless” (further reinforcing the perceptions outlined in #2 above). So whether for a lack of mandate, lack of perceived value, the complexity of discovery options, or a combination of the above - most CMDB’s projects historically produced a CMDB that fell far short of its initial promise.

The Good News 

Several technological shifts in the last decade have provided hope for the CMDB. The emergence and eventual dominance of Software as a Service (SaaS) ITSM solutions such as ServiceNow (https://www.servicenow.com/) have provided a more nimble and accessible CMDB. Modern CMDBs can rely on federated data from other trusted sources within the enterprise or the public cloud via APIs and generally available connectors. Platforms such as APM tools like AppDynamics (https://www.appdynamics.com/) and Dynatrace (https://www.dynatrace.com/) are typically funded and maintained by the applications teams and can federate trusted application relationship data to the CMDB. When paired with virtualization engines such as VMWare (https://www.vmware.com/), infrastructure data can be federated and related to the respective applications they support. 

Example of APM integration (in this case AppDynamics) into the ServiceNow CMDB 

While the discovery challenges outlined above still exist, a federated strategy that leverages platforms that are already trusted sources mitigates the value/mandate challenge outlined above, and allows already trusted data to flow to the CMDB. 

...and into the Cloud

While the advancements outlined above are foundational in helping to insure the relative success of a modern federated CMDB, the resources and practices of the public cloud may hold the key that allows the CMDB to deliver on the original promise. Public cloud adoption can be driven by a formal and structured data center exit which requires migration (push), the adoption or migration of an application driven by a business unit that then requires support by IT (pull), or the acquisition of a company that already has a cloud footprint (cross-pollinate). All of these motions drive a response from traditional IT organizations that can vary from tactical mitigation (e.g., “How do we keep the lights on with minimal change?”) to transformation (e.g., “How do we adopt full DevOps best practices in both how we build and deploy applications as well as immutable infrastructure?”). Adoption of the public cloud creates a shift towards standardization and automation that impacts the CMDB initiative in several positive ways. As organizations adopt best practices within the public cloud, federation, automation, and tagging strategies are starting to provide many of the insights the CMDB always promised. 

Amazon Web Services (AWS) defines tagging as the following:

“Amazon Web Services allows customers to assign metadata to their AWS resources in the form of tags. Each tag is a simple label consisting of a customer-defined key and an optional value that can make it easier to manage, search for, and filter resources. Although there are no inherent types of tags, they enable customers to categorize resources by purpose, owner, environment, or other criteria.”

Since tagging works foundationally at the resource level, every compute resource can be tagged to provide contextual information with a CMDB. The AWS Elastic Compute Cloud resource of EC2 is the most foundational resource with the AWS cloud. An example of a standardized tagging mechanism is shown below:

Excerpt from AWS Tagging Best Practices 

Tagging minimally provides the foundation for resource management through metering and tie-backs to cost centers, environments, and applications. This can be further expanded upon to denote both financial and support ownership which might be established in the CMDB as described below:

Excerpt from AWS Tagging Best Practices 

Tagging can be done from the AWS console, but AWS and DevOps best practices dictate that tags be applied through a standard tagging strategy as part of standardized and automated deployment. 

Conclusion

Advancements in SaaS ITSM platforms and public cloud technology can help insure the success of the CMDB in the following ways:

• Automated infrastructure provisioning and application deployment based on DevOps and public cloud best practices enforces tagging strategies that ensure valuable CMDB attributes are systemically assigned to resources essential to application service levels.
• Federation of APM tools can establish upstream and downstream impacts on a near real time basis, from sources trusted by the application teams. This alleviates some of the burden previously shouldered by traditional IT teams, and helps provide the level of context and confidence required by a “trusted source” like a CMDB.
• Automated infrastructure provisioning and application deployment allows for automated instrumentation of monitoring and logging of applications and infrastructure (e.g., CloudWatch and CloudTrail in the AWS public cloud example), which can correlate events and log data to resources tagged in the CMDB.

There are of course still challenges and reasons for caution. Nothing absolves organizations from having the proper mandate and stakeholder engagement when implementing a CMDB. “Lift & shift” cloud migration and console based infrastructure and application deployments are still prevalent in the public cloud, and there can be a substantial organizational change commitment to adoption of DevOps and public cloud best practices. But the argument can be made (indeed I believe I am making it here) that if the organization is not committed to the best practices outlined above, then a CMDB project is not likely to succeed. That being said, the advances in readily available SaaS options and architecture, combined with what is an inevitable trend towards public cloud and DevOps best practices are converging to make this perhaps the best time ever to implement an effective CMDB.  

* The author acknowledges that the ITIL definition of the CMDB is far more expansive and holistic than I have described here, but I have reduced the argument to what I believe to be the core value proposition of the CMDB, independent of IT Asset Management (ITAM) or the other service based attributes. 

References:

IT Service Management Definition: https://en.wikipedia.org/wiki/IT_service_management

IT Infrastructure Library (ITIL): https://en.wikipedia.org/wiki/ITIL

Configuration Management Database (CMDB): https://en.wikipedia.org/wiki/Configuration_management_database

Application Performance Management Definition: https://en.wikipedia.org/wiki/Application_performance_management  

AWS tagging strategies: https://d1.awsstatic.com/whitepapers/aws-tagging-best-practices.pdf

DevOps Definition: https://en.wikipedia.org/wiki/DevOps