Subtle data implications of microservices

Meetup groups, LinkedIn, networking in Raleigh, NC . . . I’ve been discussing some of today’s interesting trends in software and information technology. I’ve synthesized several discussions and their tangents for this article. On behalf of those that spoke with me, and myself, we hope this insight helps you.

Microservices offers compelling benefits to reduce the cost, risk, and complexity of software changes. However, don’t expect your data management woes to improve. If your data architecture, data management, and data quality lacks rigor and insight to inform decisions and corrective actions today, microservices may be lethal to your organization.

Martin Fowler describes microservices architecture by stating, “The term "Microservice Architecture" has sprung up over the last few years to describe a particular way of designing software applications as suites of independently deployable services. While there is no precise definition of this architectural style, there are certain common characteristics around organization around business capability, automated deployment, intelligence in the endpoints, and decentralized control of languages and data.”

An alternative definition I’ve concluded from my discussions is, “The term microservice describes whatever a collective group solving a problem using the term declares it to be”. Sigh! But term abuse is an article for another day.

For many organizations, microservices will be adopted as a strategy for refactoring monolithic systems into several, or many, smaller systems with the goal of simplifying the complexity, cost, and risk of change—all which complement other ongoing trends across corporate IT—adoption of an Agile (or iterative) methodology and the Cloud.

I’ve spoken with functional leaders, architects, engineers, product developers, managers and other Agilists. I try to understand what the company’s, or individual’s, experience is and what problem they believe will be solved by adopting microservices. The problem often begins with the story of success at Amazon, Hulu, Netflix, etc., and proceeds with discussions about Agile, dependencies and bottlenecks, and moving to the Cloud.

The focus is consistently on strategic systems—those being invested in for the foreseeable future—and not those systems that are intended to be decommissioned in the foreseeable future.

To provide context, and understand scale, a diagram helps architecture discussions. Rather than craft a unique mental picture with each discussion, I’ve been using an analogy. The map of the United States of America will represent a company’s view of their enterprise architecture. Since most people have a concept of this map, it works, even globally.

In our analogy, each state represents a monolithic application. As is the case in enterprise architectures and in the United States of America, no two states, or two systems, have the same shape (features), size (scale), demographics (technologies), population (data and data volumes), economy (cost structures), etc.  

With an agreed conceptual understanding and analogy of their portfolio, the discussion continues with details about how the journey begins. Because I like to understand the decision-making process, I ask “how, why, with what” questions, and eventually look to visualize a high-level roadmap, with guesses to fill in missing details. Not surprisingly, there’s discussion about adopting open source technologies to exit costly commercial licenses, as well as retiring components by adopting new cloud services. The architecture and planned use of technologies makes sense.

The future’s going to be great! And when they succeed—and why shouldn’t they?—their enterprise architecture is going to be very different: it’s going to be granular. Each service will be smaller and less complex, with fewer features, few dependencies, and automated testing and deployment. It will be less costly and risky to change and operate; therefore, commercial rewards await. 

But what happens when we peek into the future? Are there less complexities and fewer dependencies?

Continuing our map analogy, and not applying a one size fits all approach, we’ll assume that each monolithic application, a state, has been refactored in dozens, or hundreds, of microservices. Seems plausible, so in our example, the new microservices correspond to the counties in each state’s map.

Hypothetically, if your architecture followed this analogy, you’d have started with 64 monoliths—50 states and 14 sovereign territories—then refactored them into over 3,000 microservices. 

This isn’t an argument for or against either approach in the abstract. What’s important is to understand that there are differences in the commercial benefits, implications, and tradeoffs of each, and the underlying IT operating model to enable them.

3,000+ systems! I’m intrigued. I’m impressed with their command of contemporary architecture and technology; frankly, at times even envious. Let’s face it: it’s a very exciting time to be working in software and information technology. But does the excitement mislead? Is the focus on technology for technology’s sake? Or, are compelling business problems that deliver tangible commercial benefits being solved?  

But what about the data, I ask? No, not that big data buzz. Just plain old data. The very data the monoliths store, process, and transmit today. The data that is the raison d’être for these systems. Perhaps you have been in one of these conversations? Perhaps you’re a customer of a company like this? Perhaps this is avoidable?

Microservices amplify demand for sound data architecture, data management, data quality, and data governance programs.

When I’ve consulted with my friend and data management guru Ted Hills, who’s been in too many of these conversations judging by his response, he replies (paraphrasing), “It’s great you’re replacing and modernizing all those systems! I’m curious, are you replacing all that data too?”

Meanwhile, here are some things to consider on your microservices journey:

• Does your organization have a data governance program? Who is the executive sponsor?
• How many data architects does your organization have? Solution architects? Enterprise architects? Is it working?
• What is driving your scoping decisions for microservice boundaries? Are you familiar with Domain-Driven Design?
• What are the implications of shared-nothing infrastructures on the re-architecture, design, availability, development and operations, and costs?
• When databases are not shared, how does the data get to each microservice?
• What are the business implications of eventual consistency?
• Sarbanes-Oxley, Basel I, Basel II, HIPAA, GDPR, PCI-DSS, oh my! What are the implications of increasing the surface area of sensitive data throughout the microservice ecosystem? What about test data management for same? Bonus, who’s problem is this if something bad happens? (Hint: probably not the CISO.)
• How is Architecture Knowledge Managed? For example, do you know the technologies / versions used in each of the 50 systems today, without a fire drill, so that you can proactively respond to security alerts? Is the data accurate and efficient to steward? Will the capability support 3,000, or more, microservices? Will it support the team autonomy Agile demands?

In summary, I agree with the notion of microservices as an architecture strategy to increase granularity, and therefore have a favorable effect on the risk, cost, and complexity of changing the grain. However, until microservices achieve the maturity of their vision—being independently deployable—the complexity of the whole increases—significantly—and potentially beyond today’s complexity. Enterprises need a comprehensive functional and data architecture to inform the journey. There’s also an enormous need for process excellence without bureaucracy, knowledge management, tooling, and—most important—committed talented people. Without these, it’s magical thinking and change for technology’s sake.

Enjoy this article? Why not connect, or follow me?

Images used with permission from www.freevectormaps.com.