Micro-services and Test Environment Automation

An interesting development I've noticed in software engineering is the trend towards breaking up the components of the project into micro-services. This topic recently motivated me to write some software to deal with some of the management overheads from the approach. Feel free to make use of my code if you have similar problems. https://github.com/JohnShield/scriptclerk

I feel micro-services comes from the general good software practice thinking where code is modular with well defined interfaces. However, there's some drawbacks too with the micro-service approach.

The positive is when it is done well, you end up with toolkits like the Unix operating systems. The engineering team can develop the system concurrently and maintainability of the system is high. You can also get some performance without thinking hard on threading as the approach leads to some concurrency. With strongly defined interfaces it can be easy to figure out problems in the system as the responsibilities and errors can be made clear in the interfaces between the various micro-services.

One drawback is management overhead from having to deal with starting up and configuring multiple services. That's what motivated my little software project, as the configuration and startup overhead is even more painful during development. You need restart the software many times if you want to test interactions for each feature as you develop it.

An observation I've made from my last two projects where the micro-service design pattern was used, it increased the speed that features can be created as the software can be more loosely coupled as you don't have to test how it works in the larger system. However, it slows down the later software life-cycle stages during integration and testing, because late stage problems that come about from the interactions between the micro-services become more difficult to see and track down. Bugs that become obvious very quickly in a more integrated system are often overlooked due to testing of the micro-services done in isolation. It can also lead to fragility in the code as it becomes easy to demand perfect behaviour from the connecting services, and make no allowances for problems or error states that invariably happens. With that design philosophy, a single small bug can propagate easily and cause multiple services to fail at the same time making tracking down the original cause very time consuming.

I think using micro-services requires more discipline to get right, because it makes the project more rewarding and quicker during feature development, but more difficult to finalise to a high quality. There are a few ways to mitigate this particular problem. Firstly, having the discipline to develop well defined interfaces early. Secondly, avoiding the poor mentality of expecting ideal behaviour external to the module, which is an incorrect mindset that comes from taking software engineering theories of modularity too literally.

I'm a fan of the micro-service approach early on, because it makes the project more satisfying for a developer with quicker feature development. However, I definitely dislike the later side effects of increased difficulty in trying to get a stable system.