Data-driven, data-informed or data-blocked?

Disclaimer:

• This article is purely based on the author's limited observation and experience. It might not fully reflect the entire industry practices. Therefore, the recommendation might not be relevant or useful for all businesses.
• All the examples, including the events and people involved, are entirely fictitious. Any resemblance to real persons and/or companies is purely coincidental.

Oh, data, my precious!

For the past few years, data has become a new fancy word of aspiration for many companies and job seekers in Vietnam.

"Data is the new oil."

"Data scientist is the sexiest job of the 21st century."

Yeah, no one can resist the omnipotent power of data. Suddenly, businesses realize that they need a team of data experts. 

Okay, where is our business intelligence team? Of course, we need a financial analyst, and also a marketing analyst. Oh, we also want a team of data scientists to add artificial intelligence (AI) to our products and/or services.

The data culture

So is data that all-mighty? Is it the silver bullet to solve every business problem? Is it the north-star for your strategy? 

Well, maybe, maybe not. Data is literally just numbers or fancy charts (actually, it can also be photos, videos, audios, and others but that's another story). How effective it can help the business majorly depends on the ways their people can leverage the data to drive their decision-making process.

How to make that work? We will need to improve the data literacy of the business people and build up a data culture access the company. 

Then here come the buzzwords of "data-driven" and "data-informed".

Data-driven vs. -informed

Okay, fancy words ha. So what are the differences between them?

• Data-driven: always relying on data to make decisions and that each element of the business strategy is based on interpretations of data. Data-driven doesn’t take personal experience or insight into account and focuses less on the bigger picture. It’s simply about the cold, hard facts. In this approach, data has the final say. (*)
• Data-informed: using data alongside people's unique experience, user research, and other inputs to make decisions. With a data-informed approach, data is just one part of the decision-making process. (*)

Here I found a cool illustration by Chanade Hemming.

Potential problems with the data-driven culture? 

In the data-driven approach, data makes the call. However, can we be sure that our data is perfect? Is it incontrovertible? Errr, probably that's a no for most of the time. 

Any data can have its own blind spots (**). Without a clear understanding of the number or a reasonable way to address its deficiencies, we are more likely to make sub-par decisions.  

On the possible issues with the data-driven culture, I would like to share this very interesting article by Uzma Barlaskar: Why you should be data-informed and not data-driven

From the reading, we can get the 3 main problems one might face when being too data-driven:

1.We might apply insights from data literally without thoroughly understand the behavior that’s behind the data.

2. The data we measure might not accurately capture the behavior, including:

• Leading vs lagging indicators: more than usual that we only measure the lag metric. When we get the numbers, it's all the past and we can do nothing to fix them.
• Impact of participation vs volume: a small group of power users may be biasing the results and the large majority of the users are impacted negatively.
• Impact of context: when making a conclusion without a proper context doing a comparison between two different datasets, we are very likely to get biased insights.

3. Trends are changing: since early trends rarely show up in the data, this can be a blind spot for most large organizations and is the space, where startups thrive in. Disruption of large organizations happens when they ignore or don’t notice early trends, and instead rely on existing data

Apart from Uzma's points above, I also want to ask a few more typical problems in the early stage of building a strong data culture:

Over-extrapolation:

When it seems to be the norm to get everything backed by data, it's now extremely stressful for anyone coming into a meeting with no numbers on their hands.

Solution? Extrapolation. You pick what we know then find a way to turn it into what we want. For example, we need the forecast for next year's revenue. We can take yesterday's revenue then multiply it by 365. Tada!

Hold on, it doesn't work. We don't know how the market will look like for the next 12 months. We also don't take into account the seasonal fluctuations or other external impacts. How can this number be trustworthy?

Source: xkcd

Extrapolation is usually subjected to greater uncertainty and a higher risk of producing meaningless results (***). By over-extrapolating, we are more likely to make a mistake than a good prediction. 

Over-fitting:

Unlike extrapolation that drives the numbers into the unknown, over-fitting would be a bad result from an effort to sharp the data into an expected outcome.

Source: xkcd

Say we did a forecast for the next quarter. The number we have is 500 Mio. However, the company target is 7 Bio but the forecast is just around 500 Mio. Hmm, too low, what to do?

Let's add a multiplier called "Covid recovery factor". Things will surely be better next three months, right? So we would expect a double-digit growth rate. Does it sound good? 

Errr... not really. One can calls it estimation but I would prefer miscalculation. 

Neglection of the Confidence Level:

It's the sad truth that you can hardly ever get a perfectly accurate number. There could be outliners. There could be a limitation in the sampling method. There could also be missing data. That means every number has its own confidence level.

More than usual that people just care about the numbers but not their trustworthiness. An estimation with 50% confidence is not very much better than a guess, right? The problem would be even worse if we chain several numbers with a low level of confidence together. 

Like we try to estimate the yearly income of a banker. First, we get the average monthly salary of a typical banker is 10 Mio. The range would be from 8 Mio to 10 Mio. Then we assume that this banker only gets a 12-month salary package, not likely a 13-month one. Last but not least, if he is an excellent performer, we can get an bonus of up to 4 months' salary. Putting everything all together, the income of this banker ranges from 96 Mio to 170 Mio. 

Now the range is too big, it hardly provides any valuable insight for us.

A bigger issue the over-data-driven culture: being data-blocked?

While a data-driven culture helps to equip business people with facts and figures to back up their decisions, it could backfire and create a strong dependency on numbers.

What is even worse than that? Data become a blocker that hinders our potential. 

"We will not do [the project] until we get [a number]"

"If we can't measure [a metric], we won't process further [an initiative]"

Are those statements familiar? If your answer is yes, well, we have not-so-good news here. It's now a data-blocked culture. Everyone asks for more data rather than actionable items or deliverables.

Making decisions with little to no data is very tough but we have to keep moving. That's how business should work. Our competitors don't sit and wait until we have the sound numbers to act.

My previous boss likes Mark Zuckerberg's famous motto: “Move fast and break things.” I do agree with him. The ability to take bold actions despite uncertainty is way more important to a business. Things can go wrong, even if you have all the numbers well put. That's why you need risk management and a contingency plan, right?

Conclusion

Data is still the new oil and data literacy is still the next important skill set for all of us. 

However, while embracing the data culture among your company, beware of over-data-driven behaviors that set back your decisions. Data should be there to streamline your business, not paralyze it. Does this make sense?

---

(*) Data-Driven vs. Data-Informed: Why You Need Both Approaches

(**) Why you should be data-informed and not data-driven

(***) Extrapolation