Why do I need Data Analysis AND Test & Experimentation teams?

All analysis must be tested. All tests must be analyzed. The twine cannot be separated - one cannot live without the other. And yet, far too often, we see Analytics teams not having the supporting Test & Experimentation support built in, and the Test & Experimentation team not provided the necessary depth of analytical support it needs. The best data companies of course usually have it right - these two functions work together, arm in arm, generating valuable insights for the business.

Businesses, whether they say it loudly or not, are often in search of causal analysis - to properly understand whether fueling activity A, will result in outcome X? Alas, pure data analysis can only go so far in getting to causation, except in some unique circumstances. Correlation, Association and such - yes, data analysis shines there, and often with a fairly high degree of confidence. But understanding Causation usually requires an excellently designed experiment, that controls for several factors and checks for true cause. This is not easy, but is one of the closest things we have for getting to causation.

Well, then, why not keep doing experiments only? Usually because experiments need a longer set-up, and longer run-time. Data Analysis is faster and can cover more ground and can point towards tests that ought to be run. It thus acts as a phenomenal filter that can guide the Test & Experimentation world. Not to speak of post-test analysis - which is not always straightforward. Sometimes the results themselves are very surprising, and needs deep data analysis to understand it.

Then why are organizations evolving where these two functions are not placed together? Reasons are wide-ranging, including:

• The institutions behind these functions in the market have evolved differently - Statistics, Machine Learning on the one hand & Digital Marketing on the other
• Even within an organization, Testing & Experimentation is seen as a "digital" activity while Reporting, Analytics, Data Science are seen as an "operations" activity
• Absence of leadership talent that understands both these aspects well enough to pull this off under a common umbrella
• The industry Analyst community has not done a good job in trying to coalesce these activities together either

As stand-alone functions, they still deliver value, but always leaving something to be desired. To quote professor Clayton Christennsen - "A great book seeks to explain causality, not correlation". Similarly, a great Data & Test group does not rest until causality is established. And when a series of causative insights are generated for the business, then, and then alone, will we be able to say with conviction, that if the company does action A, it will yield result X; and if it does B, it will yield result Y.

And the business can move forward.