Breaking the Monolith
Lots of articles have been written about the topic of microservices and ways of transforming legacy large enterprise platforms into scalable, performant, manageable, granular applications and services. Microservices is not a new concept, it's just a new term applied to proven software architecture and design patterns. From components to SOA to microservices, they all have plenty in common, specifically ability to separate concerns, reuse, scale, manage, deploy without resetting and changing the world.
With that in mind here is what I found important when the decision has been made to "break the monolith". And believe me, sooner or later this decision will definitely be made if you want your business to grow, evolve and stay agile. There is a lot more than technology decisions that have to be considered, but from purely technical point of view, everything you do will have to revolve around IaaS/PaaS (e.g. AWS), containers (e.g. docker), events/messages (e.g. Kafka, ActiveQM, RabbitMQ, SQS), horizontally scalable databases (e.g. Cassandra, MongoDB), Single Page Web Applications (e.g. Angular, React), CI/CD (e.g. Bamboo, Jenkens, Ansible, Chef, Puppet) and an open, agile, lean, DevOps culture.
Here is a high level process:
Discovery
- Document all of the existing business processes, web, backend and messaging components. Doesn't have to be a 100 page document. High level overview is enough at this stage in order to assess the complexity of the existing platform.
- Document goals of digital transformation. Why and what are the important questions to answer.
Assess your development team's strengths (programming languages, databases, tools, frameworks)
- Are you a RoR/Postgres shop? Java/Spring/Oracle? PHP/MySQL? Python/Django/MongoDB? The list goes on, but the goal here is to understand that if you have a good development team with solid knowledge in specific stacks, it makes sense to base your target architecture on those skillsets. Introduction of new technologies will be inevitable, but should be minimized to backend databases, tools and frameworks, and not core programming languages.
Assess your ops/infrastructure team's skill set and experience
- Most legacy platforms are managed by traditional operations/infrastructure teams, separated by a brick wall from development teams. This is just the way things are. Companies are slowly moving to DevOps culture, which means different things to different people, but the goal here is to make sure to identify the gaps within current it/ops and to gradually fill them via mentoring/training and new hires.
Research microservice frameworks that are most widely used with your team's existing technology stack.
- If you're a Java shop then go with Java/Spring Boot. For JavaScript go with Node.js. Python, .Net, you name it, each stack has good microservices frameworks. Do some research, build a POC and get your team excited.
Research JavaScript frameworks.
- It's very important to understand that the traditional monolith, server driven MVC style web applications do not scale in the new world and are anti-patterns for microservices based architecture. SPA (Single Page Application) is the way to go, with separate deployments of web applications from all microservices. Angular, React, Amber are a few popular frameworks to choose from.
Decide on on-premise vs cloud vs hybrid target infrastructure.
- It's not easy to just move to AWS or Azure if you have existing large on-premise infrastructure. Decision on what goes to the cloud vs what stays on premise has to be given serious thought. Take following into consideration: security, scalability, fault tolerance, cost, team skill set, ease of deployments.
Design high level architecture.
- Ok, here is where the fun part begins. How do you want your super-duper microservices based platform to look when it's all done (let's say 12 to 24 months from now) and why? How can you demonstrate the benefit of a large digital transformation initiative in 10 to 20 powerpoint slides and architectural diagrams and get full support and buy-in from the C-level executives? Well, make it balanced with the right level of business benefits, required investment, research/comparisons and of course sleek diagrams. You will have to go through multiple iterations, with multiple stakeholders until everyone (or most) are happy with the direction. Word of caution - there would often be resistance to change and even when the initiative is on its way there will be those who would enjoy to see your hard work go up in flames. But you're not going to let it happen!
Create POC for any technology not currently used by existing technology team.
- Let's assume that your current development team is using Java/Spring MVC, Tomcat and Oracle RDBMS and your target architecture consists of following technologies (AWS, Java, Spring Boot, SQS, SNS, ECS,Angular, Cassandra, Solr, Docker, etc). Here is where you need to make sure that development team understands all the moving parts, how they are built and how they interact with each other. Clone some repos from GitHub, create sample services, build docker images, deploy to AWS, hit some /hello endpoints, get the developers excited!
Partition architecture into "framework candidates" - parts of the architecture that can be developed as frameworks to simplify and standardize future development effort.
- Some of the obvious candidates are: Core (common db config/access, utilities, rest clients, security, logging, monitoring), ETL Framework (generic way to move data from legacy database to new database), S3 Storage Service, etc.
Select first application to develop using new architecture.
- This should be medium complexity new application. Not too simple and not too complex. Should preferably include most components of the target architecture (web application, microservices, events, etl, database). The goal of this first project is to show without any doubt that the new architecture is solid and make it an example and a reference implementation for all future development.
Break the monolith. This deserves a detailed separate writeup, but let's stick with high level for now.
- Document future detailed architecture ( web applications, microservices, interactions, security, etc).
- Prioritize refactoring effort.
- Prioritize new services/applications effort.
- Adopt agile process.
- Involve DevOps from day one.
Have fun! Really!
----
"Life is a series of natural and spontaneous changes. Don't resist them - that only creates sorrow. Let reality be reality. Let things flow naturally forward in whatever way they like."
-Lao Tzu
----
This is a great article! However, one of the key elements of digital transformation is the cultural change that it forces in the organizations. I think often times that's a miss. It's dangerous to see digital transformation just as a technology implementation, as it has significant impact on people.
Hi Igor, I thank you for effectively covering a lot of ground with respect to the technical aspects of digital transformation in this brief write-up. You've rightly emphasized the complexity of change management. What experience taught us is that an organization's capacity for change greatly varies by the clarity of the vision, buy-in into stated goals, and effectiveness of the operating environment. For example, when initiatives to break up the monolith lack clear and balanced goals and accountabilities (customer, financial, execution, continuous improvement), the void is filled with beliefs, conjectures and theories, which eventually stall the progress. Another point to highlight is that not all parts of monolith will be broken into micro-services. There is a great deal of executive level risk/reward trade-offs to be made to avoid over-engineering, and conventional wisdom of reuse-standardize-simplify may mislead us. There is a great lesson from the history of manufacturing industry: During the heydays of mass production, the conventional wisdom was that push-systems and scale economies were two must-haves for automobile manufacturers to be cost competitive. Then, Toyota invented a new methodology and proved everyone that they could produce each part cheaper with pull-systems and much smaller batch sizes. This was the the turning point in the manufacturing industry...
I've worked on many transformations, after they've fallen into dispute. Be wary of being pushed down a slot based delivery process. As a customer, you may consist of many businesses/offices/teams running different systems/apps requiring more/less resource led data migrations. If you agree to a number of slots, you could find they've all been used up and many of the workforce are still not transformed.
Igor Royzis - Thanks for sharing the process in your article above. Can you please share some success stories or case studies for the organizations that have successfully implemented the digital transformation you described above? It would be great to learn the before and after changes and the impact on the organizations. Thanks
Igor, share some thoughts on Transnational Data in Micro Service world, how to achieve ACID properties when dealing with multiple micro services to achieve a business flow. I understand there will certain use cases those will not be Micro Service candidates.