Software design, part 5: How to grow software to large scale using modular design and evolutionary architecture.

In previous articles I was describing design methods or patterns that, when being followed, will lead to better software. I was not describing a framework. Modular design is not a framework and not a pattern that you simply use or set-up at the beginning and will do everything by itself. It is more of a mindset. Modularization is the first and big step, but it is far from being over. Many more steps need to be performed when software grows.

The problem of many software and also a reason why it in many times fails is the lack of design when it grows. It usually starts nicely, using some pattern, everything is nicely structured and clean, but we all know that later on, requirements usually change, new functionalities are added that were not planned, etc. Developers start maybe with a hack here and there, it needs to be done fast so simply adding some lines won't hurt. The software was maybe designed in such a way that new unplanned functionalities just don't fit into the design and workarounds need to be implemented in order to make it work. Then come again new requirements, or many small things that need to be added and so the software grows, hacks, shortcuts and bad code pile up until they quickly bring software to being a monolith where everything is mixed up with lots of spaghetti code. I've seen many code and original structure that was intended for good organization like MVC, MPP, MVVM, N-Tier, then with structures that were meant to make things easier to use, but later on during development, classes with 2000+ lines of code showed up, functionalities mixed and placed all around that find a place to add new code to was a nightmare.

This is exactly what modular design is solving. By keeping functionalities together (high cohesion) and building dependencies with DI and DI Container, you can more easily see how software works and how the are things wired together even if it is not written nicely. You'll be able to manage it better. This will prevent developers to just simply and quickly add some new code anywhere. Developers need to think, into which module should they put it, or create a new module. It's a question of sense and developers need to ask themselves constantly "Does this make sense?" during the development. That's why modular design is an continuously ongoing, not just one-time process.

Here is a recipe on how to start writing software and grow it with modular design in mind.

Start minimal. Modular design doesn't require any upfront design. When you know what functionalities you need in software, simply create a place to start putting your modules to. That's it, 5 min of work and you're set. After that, I always add bootstrapping code to initialize the DI container which takes another 5 min to do. This is a base, no big architecture, nothing. Now simply put your first modules in, connect them together and run the application.

New functionalities are added in exactly the same way. They are simply put in the same place where other modules are. Modules are now organized in a simple list. If your application will not be big, then you can leave it like this. If your functionalities are very small and will most probably stay small, you don't need to create separate assemblies (in .NET) for them. Try to design modules in such a way that they are expandable that when you're adding new functionalities, they are attached to existing ones without having to change them (or are changes minimal).

If the list grows large, then you need to start with higher-level organization. These are then the next steps that I was talking about that you need to take after the first modularization step. Now I cannot give you exact instructions on how to perform the next steps because there are many ways it can be done and many paths will successfully lead to the result. What I would do is group modules together that somehow fit together. With this first level of grouping you can now grow your software further. When this gets too big, I would do second-level grouping. We're basically building a tree structure here. In a large software the number of modules will be huge and this organization is crucial.

Growing software even bigger to enterprise levels opens again new options, for example to split software into bigger blocks that may even run on different computers and communicate through some API, message queue or enterprise message bus. This is then similar to SOA. But keep in mind, when growing big, don't forget small things. You will still have (and have to organize) smaller modules when you zoom in.

The path from small to big software also involves refactoring and evolution of the modules and structure. Modules will grow by number and size. When becoming too big, will be split into many modules, or they will die because functionality will not be needed anymore. Code in modules will be refactored and internal module structure will grow in exactly the same way. Modules are also not tied to a specific location and they can be moved around to change the structure during development. This evolution is required and is driven by changing and new requirements. Because up-front design is almost non-existent and the structure grows in time, it is an "evolutionary architecture". This term is being mentioned with micro-services but in my case, it applies to modules. Term "evolutionary architecture" has a big meaning and big impact. It fits amazingly well with agile development because we know the least about the software at the start and things will get clearer during development. Agile software development process as we know it deals with developing software through time on a functional level while modular design deals with developing software through time on a technical level.

When software grows big it is in any case getting more difficult to navigate and find things. Modular design does make things hugely better and structured but there are additional techniques that can improve it. My long-time idea was to be able to visualize the software structure by simply generating it from code. This is actually possible because we already have everything we need in the DI container. We have a list of registered classes and we know the dependencies, so why don't we use this information to generate a graph? Well I did this and to find out more, check out one of the next articles.

Modular design opens one more interesting possibility and that is writing imperfect software. Usually we want software to be nicely structured, with pretty code and perfect. Why? Because we don't want bad code to spread and lose control which grows into spaghetti code. But modular design actually allows you to write code that is not perfect, that is a bit quickly written and a bit ugly. When such code is isolated in modules, it by-definition cannot grow outside of it, providing that at least the interface to the module is clear and well written. This method actually speeds up development because developers don't need to spend time on making code perfect ("good-enough" architecture). Perfect software costs more and modular design makes imperfection manageable allowing you to develop software faster.