Understanding technical debt

So we all get behind at some point. We do it as individuals and families, we do as companies. Sometimes we get behind because of unforeseen circumstances, other times we are just bad planners. At times we do it consciously knowing we will have to deal with it later. Debt tends to creep up on us, leaving us with the sense of “How did it get this bad?”.

One of the current buzz-phrases is technical debt. For most people this equates to technical work or infrastructure replacement that we have been putting off that is now catching up with us and has to be dealt with soon or there will be consequences. Our servers are now eight years old because we lost track of their age, or we decided to upgrade the firewalls first, or…well you get the drift.

The problem is that this is not a text-book definition of technical debt. Technical debt was coined as concept in development, and relates to refactoring of code. In essence what you do is make a decision to put something ugly into production in the interests of time to market, knowing you are building technical debt that will have to be dealt with later. The cost of refactoring is the debt payment, and it is a consciously made decision.

That is the kicker – real technical debt is planned, and part of the planning includes how the debt will be paid back.

I once worked for a company that sold a “product” to a client as part of deal, but the product did not exist. The assessment was that the deal was significant enough that we would commit to building the non-existent product which we could then reuse later, thus effectively modifying our roadmap to get a large deal. I won’t deal with the obviously ethical questions here of whether the client knew that the product was not real (buyers beware, vendors do that).

When the product had to be delivered, we had nothing three months out. Product Development had spent a year digging their heels in saying they had not committed to it, while the client spent the year warning they had a government mandated deadline they had to meet. Finally two months before delivery, there was the typical vendor fire-drill, and developers where put on the task of delivering a minimum viable product.

The decision was made to deliver something that looked fairly decent with large chunks of missing functionality that was architecturally ugly to make sure the client deadline was met. In a sense this was the true definition of technical debt. The product was delivered kind of on time, the problem was that decision did not take into account the payoff. The fact that it kind of worked now became a factor. Be careful what you put into end users hands, if it looks decent you may have to pry it from their cold dead hands, and now there may not be much of incentive to go back and revisit the refactor project.

Many data centers find themselves in a technical debt situation that is not true technical debt. Here are some ways of dealing with decisions that could result in unintended technical debt up front.

1 – Life-cycle planning – It is amazing how many organizations will have product managers for solutions like their EHR, ERP or payroll system, but nor for their infrastructure. We do regular upgrades on our software (if we are doing our jobs), why would we not be planning for infrastructure upgrades. Industry standards are fairly simple to understand, but very often we do not do infrastructure life-cycle planning until the equipment is end of life.

2 – Total cost of ownership. If you do real TCO planning you are automatically building life-cycle management into your products, even at the solution level. A true TCO should not just include software license costs, software upgrades, support etc., it should include the regular upgrading of all the components required to support the system, right down to the frames and firewalls. Oh and bye the way, remember costs increase year over year when you do your TCO.

3 – Education at the C level. The C suite will often expresses frustration at the seemingly endless parade of unplanned new requests for equipment, consulting, and end user devices. Very often when I have spoken to organizations about how they got into unplanned technical debt, I am told that the C Suite does not understand why the upgrades are critical so they turn them down.

Firstly, I don’t buy that. Any C level person who does not understand that buying technology is a roller coaster you pay for forever should not be in the position. More often I find no-one has ever sat down for a few hours and gone through the implications of the technology strategy you have embarked on with the CEO, CFO and COO.

Secondly I have never met a CEO who say they would rather be left in the dark about pending expenses and deal with it later. For some reason technical folks assume that the C level wants to be shielded from the impact of technology. They don’t, what they want is correct planning, for the implications of a technology decision to be clearly spelled out (including the financial implications, and for a true TCO to be developed, because without it there revenue projections are almost certainly false.

4 – Stop chasing the bright shiny object. This of course is a double edged sword if the bright shiny object is the future, i.e. not chasing the bright shiny object does not mean do not invest in current or emerging technology. It means that if you have a choice between new projects and keeping up with maintenance on existing systems, do that in a holistic planned manner.

The analogy is so obvious in some ways it does not bear mentioning, but I was asked to help explain to one company why they had ended up in the situation they were in, with a technical backlog of years of years of server and technology upgrades. I pointed out that for any of us who owns a house, we all learn at some point the true cost of home ownership. If we did not factor that in when we worked out how much house we could afford, we end up with the house either falling into disrepair, or we or we end up paying more for the house than we can afford.

If your air-conditioner needs to be serviced every year you can get away with not doing it for one year if you know you will definitely do it the next. Consciously putting that off year after year means at some point it all catches up, to the point that you may ending up losing your home. If you had no idea that your house required upkeep, that is a question of maturity. Part of avoiding unplanned technical debt is growing up to be a mature organization and planning (like grownups do).

Finally, if you are in a senior leadership position, either in your technically department or in the broader organization, don’t blame the person who brings the technical debt to your attention, it was your job to be on top it and understand the costs and implications for you organization. If that person kept quiet about it for years, find out why and make sure it doesn’t happen again. But if I switch on the light in your attic and show you all the junk that is there, I didn’t put there.

Technical debt does not look better with age. You can’t wait your way out of it, and the longer you wait to deal with it the more costly it gets, sometimes in a non-linear way (e.g. replace you brake pads before you have to replace the discs as well). As with financial debt it is painful, it often requires you to scale back your lifestyle for a while and re-prioritize, and is not fun to deal with it. The bottom line though is it does not go away, deal with it now or later, but deal with it.