Impact of Technical Debt in Software Development and Distributed Systems

In my previous article on technical leadership, I discussed how it can help prevent the accumulation of technical debt.

Now, let us talk about technical debt. Not exactly the most exciting part of IT, right? However, in modern IT landscapes, especially within distributed systems and rapidly growing organizations, technical debt tends to accumulate over time. Even superheroes like tech leaders cannot always prevent this. If you work in tech, you have probably encountered it, even if you did not realize it at the time. The shortcut taken to meet a deadline, the temporary fix applied to avoid a larger refactor might seem like lifesavers in the moment. But as the system grows and evolves, they come back to haunt you. 👻

Understanding Technical Debt in IT

In simple terms, technical debt refers to the cost of shortcuts or suboptimal decisions made in software development that require "paying back" later. These shortcuts might be made to save time, reduce costs, or quickly meet business goals. While technical debt can help deliver fast results in the short term, it creates extra work and complications for future development and maintenance.

Ward Cunningham, who coined the term, described technical debt as similar to financial debt. When you take on debt, you can make progress faster, but over time, you have to "pay interest" in the form of increased maintenance and slower development. [1]

So, imagine your codebase as a house. Every time you take a shortcut, like skipping a test or starting to implement with a suboptimal design, you are essentially building that house on shaky ground. Sure, the house goes up quicker, but over time, you will have to keep repairing and reinforcing it. That is the technical debt I want to discuss.

Note: For readability, the terms "software development" and "software engineering" are used interchangeably throughout this text.

What Does Technical Debt Mean for Distributed Systems?

In distributed systems, technical debt is not just an isolated problem of a bounded context or a service run by a development team. It is like a virus that spreads across your entire architecture. Microservices, APIs, databases, and cloud platforms are all interconnected. A quick fix in one service can lead to performance bottlenecks elsewhere, scalability issues, or even security vulnerabilities. Fixing technical debt in a distributed environment is complex because of the coordination needed between teams and services. Changing one part of the system can impact others, making refactoring risky and time-consuming.

And now comes the really unpleasant part 😞

That one decision to choose a "quick and dirty," "workaround," or "shortcut" solution for an urgent initiative can snowball into an entire legacy system that no one really wants to touch. Now you have to deal with a grown monolith, and every new feature or upgrade becomes dependent on it. Finally, leading to a process that is risky, slow, and painful.

Potential Impact of Technical Debt

💡 Why should organizations care about technical debt? 💡

Higher Operational Costs

Teams spend more time debugging, fixing, and adding patches rather than focusing on innovation or new features. Developers are constantly firefighting instead of building new things. Fixing bugs, patching workarounds leads to higher operational costs and reduced development efficiency.

Affecting Pace of Development

Technical debt slows down the pace of development. Instead of launching exciting new features, teams spend their time fixing what is already broken. Teams are often forced to work around existing issues, which reduces their ability to quickly adapt to new business needs or market changes.

Limited Agility

Want to introduce a new feature? When technical debt is accumulating, the system becomes fragile, making it harder to implement changes without breaking existing functionality. Agility? Not so much.

Frustration

Colleagues which are involved in the software development process can become frustrated when they spend most of their time dealing with technical debt rather than working on new features.

📚 If you want to read more about the potential impact of technical debt, please refer to the references: [3, 4, 5, 7, 8]. 📚

How IT Solutions Architects Help Manage Technical Debt?

Before discussing how IT Solutions Architects can help manage technical debt, I would like to highlight a key study: In their paper "Measure it? Manage it? Ignore it? Software Practitioners and Technical Debt" [2], Ernst et al. conducted a survey of 536 participants across three large organizations. The results of this survey revealed that the respondents identified architectural decisions as the primary contributor to technical debt so called architectural debt.

Therefore, an IT Solutions Architect should act in a way to minimize and manage technical debt by focusing on long-term, scalable and maintainable solutions. Depending on the priorization based on a solid trade-off analysis each requirement is more or less important. But Architects should be the ones keeping the ship on course, making sure you do not borrow more than you can pay back in an efficient way. So, where does an IT Solutions Architect fit into this picture?

• A solid IT Solutions Architect does not just care about the now. Architects thinking in a stratetic way few years ahead. They ensure that the architecture is scalable, maintainable, and will not collapse because of increased debt.
• Together with the tech leads of the software development or engineering teams, IT Solutions Architects should act as the guardians of clean code and design principles. By promoting best practices and ensuring they are followed, IT Solutions Architects help prevent technical debt before it even starts to accumulate. In my view, shortcuts should be identified and, whenever possible, avoided. If they cannot be avoided, they should be clearly communicated and documented in an ADR (Architectural Decision Record).
• As mentioned in my previous article, technical leadership includes implementing trade-off analyses. An IT Solutions Architect can decide when it is okay to take on technical debt (to meet crucial business needs) and when it is important to focus on the long-term stability of the system. Clear communication about these trade-offs helps keep the team aligned. But sometimes it is necessary to say: “No, this is going to cost us more in the long run.”
• An IT Solutions Architect helps drive the effort for refactoring. It is about knowing when it is time to reduce technical debt by reworking parts of the system before it gets out of control. They also understand how to do this without disrupting the business. In my opinion, the process works best when the Solutions Architect has a respectful and honest relationship with the tech lead of the development team.
• Distributed systems often involve multiple teams. For one service different teams are involved and responsible than the next. The IT Solutions Architect acts as a bridge, ensuring that everyone is aligned on the technical vision. This collaboration helps avoid not coordinated decisions that leads to increasing debt.
• Last but not least: good old guided tech/code reviews and technical refinement meetings are essential practices. These sessions help ensure that best practices are followed, potential issues are caught early, and technical debt is kept in check. They also provide a platform for collaborative learning, where team members can exchange ideas, align on technical decisions, and improve the overall quality of the codebase.

Time and Change - Managing the Inevitable

And now some more bad news: Tech debt cannot be avoided in software engineering.

As discussed in the chapter "Time and Change" from the "Software Engineering at Google" book [9], change is an inevitable part of software engineering. Every decision we make, whether to prioritize new features or maintain existing code, contributes to the already existing technical debt. When changes are made without fully understanding their implications, it can lead to accumulating debt that may impact future development efforts.

Conclusion - Tech Debt in Software Development

Technical debt can have a big impact on IT costs. Studies shows, the longer we ignore it, the more expensive it potentially becomes over time.

Xiao et al stated in [2]: "We evaluated our approach on 7 large-scale Apache open source projects and the results showed that a significant portion (51% to 85%) of overall maintenance effort was consumed by paying interest on architectural debts. This suggests that projects could save a significant amount of maintenance costs if they can discover these debts early, and pay them down by refactoring."

So, technical debt is an unavoidable aspect of software development. However, it does not need to overwhelm your systems or teams. It is crucial for organizations to implement a strategy for managing technical debt to prevent significant challenges from arising, particularly in complex distributed systems.

For IT Solutions Architects, managing technical debt is not just about cleaning up messes. It is also about preventing, identifying, and documenting it as best as possible in the first place. And by doing so, they help that your organization remain agile, innovative, and ready for the future.

References

1. Cunningham, W. "The WyCash Portfolio Management System," SIGPLAN OOPS Mess., vol. 4, pp. 29–30, 1992.
2. Ernst, N. A., Bellomo, S., Ozkaya, I., Nord, R. L., & Gorton, I. "Measure It? Manage It? Ignore It? Software Practitioners and Technical Debt," presented at the Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering, Bergamo, Italy, 2015.
3. Xiao, L., Cai, Y., Kazman, R., Mo, R., & Feng, Q. "Identifying and Quantifying Architectural Debt," Proceedings of the 38th International Conference on Software Engineering (ICSE '16), Association for Computing Machinery, New York, NY, USA, pp. 488–498, 2016. https://doi.org/10.1145/2884781.2884822
4. Besker, T., Martini, A., & Bosch, J. "A Systematic Literature Review and a Unified Model of Architectural Technical Debt," in Euromicro Conference Series on Software Engineering and Advanced Applications (SEAA), 2016.
5. Martini, A., Besker, T., & Bosch, J. "Technical Debt Tracking: Current State of Practice," Empirical Software Engineering, 2018.
6. Besker, T., Bosch, J., & Martini, A. "Technical Debt Cripples Software Developer Productivity - A Longitudinal Study on Developers' Daily Software Development Work," ResearchGate, May 2018. https://www.researchgate.net/publication/325790190_Technical_Debt_Cripples_Software_Developer_Productivity_-A_longitudinal_study_on_developers'_daily_software_development_work
7. "The Developer Coefficient," Stripe, September 2018. https://stripe.com/files/reports/the-developer-coefficient.pdf
8. CAI, "Understanding the Impact of Technical Debt," available at: https://www.cai.io/resources/articles/understanding-the-impact-of-technical-debt#fn:9
9. "Time and Change," in Software Engineering at Google, available at: https://abseil.io/resources/swe-book/html/ch01.html#time_and_change