A guide to enduring data and analytics solutions

As data professionals, we don't get the luxury of driving past a building and thinking, yeah, I helped build that. Yet deep down we all want to build things that last. Maybe not for centuries, but which stand up strongly for years in a fast-moving world.

Yet, too often our industry has a sustainability problem. Not necessarily ecologically but in developing solutions that evolve and serve effectively to a healthy age – why else are organizations re-building models and reports again and again? This hurts us all. As a profession we lose credibility and embark on a prolonged Groundhog Day, while industry itself suffers from missing the deeper value we can offer.

I recently closed off a complex end-to-end analytics project. This required extracting data from many sources, a subsequent datamart built in a new Databricks platform and funnelling daily insights to reports and a dashboard suite. I recognised this could collapse very easily. So for this to stand the test of time I set out a framework to ensure sustainability. This meaning it would survive staff re-assignment, changing data sources, system errors, diverging business priorities and many other predicaments.

I have condensed these into six key components:

Be Pragmatic

It goes without saying but we need to build the solution for the problem - not find the problem for the solution. Applications and outcomes may evolve but they rarely go away. A pragmatic focus facilitates ongoing usage and continued investment (time and money) from stakeholders. Likewise flexibility as business priorities alter (assuming effort is made to anticipate these).

An aside on this is recognizing how models/insights will be used in the decision-making process. Data insights are not 100% of the answer. Qualitative data, experience, intuition and even politics all play a part, and solutions need to work into this. Human-centred tools are much more likely to endure.

Be Lean

Most analytics projects face balancing the counteracting forces of complexity, accuracy and sustainability. Every context differs but finding the right balance is vital. With complexity comes vulnerability (just consider a modern car!). Is it worth adding that extra dataset to add 0.1% accuracy? Is that dataset reliable in quality and delivery? I would not say less is more, but lesser is often better.

Furthermore, development can be messy and broad, but when you have a solution keep it trim. Just as a building site is cleaned up, we too need to remove the offcuts and tighten the flows. A smooth stream-lined flow will perform consistently and is far less likely to break.

Be Democratic

Models should not die when the developer moves on. Too often they do. Likewise we all experience situations when someone owns a process without really understanding it. Documentation and handover only go so far, there needs to be in depth understanding of the processes and the project across teams. As my friend Alistair reminds me, this should not begin just two weeks before the builder departs! This needs to be a key part of the project plan. Standardised coding and design standards is also important. An effective strategy often used is treating all such processes as products, with all the compliance, governance, documentation and usage standards that go with that. Regardless, data and analytics is a team pursuit and that needs team ownership from the get go.

Be Adaptable

Nothing is more certain than change. The inputs and outputs of analytics solutions are no exception. Being pragmatic allows us to build around business needs, but similarly we should avoid being tightly knitted to the data or the tech. These too will evolve. By cementing a datamart design around real-world metrics (as opposed to technical inputs) we can amend code to shape replacement data into existing (secondary) structures. This will safeguard downstream models, reports and dashboards.

Reporting needs to adapt too. Adding self-service reports is great for this. But likewise, if we build strong data foundations, amending or adding subsequent reports is relatively easy. Being knowledgeable across the business will make much easier to anticipate changes and be one step ahead when these occur.

Be Monitored

Live data projects are like my two-year-old, move your eyes for a moment and they have gone rogue (or just gone!). Set-and-forget is generally unrealistic. Build in checks and alerts to proactively warn (e.g. a triggered email) when a process fails or the data is diverged from expected. The same applies to performance, particularly for statistical models. Regularly monitor that they are predicting what they should. If the data below is well built, re-tuning models should not be disruptive.

Be Fresh

A fundamental aspect of good data solution is one that is actually used. Many models and reports start with flashes of glory, but after six months the business has lost interest or been distracted by the next tool. Regular check-ins with stakeholders and users should be part of any project plan (3, 6 ,12 months+), these gain feedback and an understanding of application. Evolution and showmanship also plays a part. Regularly adapt, share results, add features. report impacts and demonstrate value. The quickest way to degradation is to be forgotten.

I don't see this as a comprehensive list, yet each of these is imperative, touching micro and macro issues. Contexts vary but aim to foresee key vulnerabilities (I like the pre-mortem approach of assessing why the project failed, in advance). Building sustainable projects is an intentional action, it won't just happen.

We won’t fully embrace (or be trusted with) the benefits of new technologies of AI and other sophisticated tools if we can’t demonstrate prolonged value. But if so, organisations will benefit from being increasingly led by insights, smarter analysis and an advancing data journey, whilst as practitioners we will evolve to more interesting and satisfying work.