Structured Data or Gut Feeling ?

I am puzzled to hear top leaders still arguing that their « gut feeling » gives a better forecast accuracy than « planners’ work ». Is one forecasting method better than the other one ? Or is « nor one nor the other but both together » the right solution?

Structured Data

Planners’ Forecast (or Structured Data) are projections of the future using past performance and mathematical models. So what cannot be - or was not - converted into a perfect mathematical model will not correctly plan/forecast tomorrow business. We understand quickly that a forecast accuracy of 50% is already a good performance in many industries and for many products.

Gut Feeling

Gut Feeling is the result of a brain process that, in an instant, or during a meeting, integrates all our knowledge and uses all our analytical capabilities to formulate an opinion. The process is so complex that our brain can verbalize the outcome while the brain cannot verbalize the rational process used (to provide this outcome).

The brain outcome - or our gut feeling - was able to integrate the « Unstructured Data » that the planner could not. It was not only able to integrate past performance but also the consequence of projected activities that are more or less certain. Gut feeling performance of a seasoned executive is undoubtedly be more precise than the gut feeling of a junior planner.

Unstructured Data

Gut Felling is one of the numerous Unstructured Data that we can use. Unstructured Data also include the unexplained sales patterns discovered from planner’s intuition and the information received by marketers, clients, retailers and sales team. 

When to use Structured Data or Unstructured Data?

The weight given to « Structured Data » rather than to « Unstructured Data/Gut Feeling » depends on many factor. We all know that the importance of Unstructured Data will depend on the life-cycle of our product as simplified in the below chart. But other criteria are also important.

Runner, Repeater, Stranger

We learned to classify our products in Runners, Repeaters and Strangers. Structured Data for Runners will usually provide a good or excellent forecast accuracy. But if one Runner represent an high percentage of our total volume, unstructured data should weight in our decisions. While for Strangers, Structured Data will be less useful.

Aggregation Level

This can be completed by T. Wallace work: more we aggregate the data, more the Structured Data will become significant.

Emotional or Rational purchase act

Another important question is to value the purchase act of the consumer. Is it an emotional decision - at a special moment as for Valentine Day - that drives the purchase act? Or Is it a rational purchase for a daily consumption  as for drugs (as in Pharma and Tobacco industries) or for commodities?

More the consumer act is rational, more Structured Data will provide a sound forecast accuracy. While for emotional purchase act, the importance of the marketing campaign, the business relation (in B2B businesses) and the  competitor’s price campaign will influence the outcome. Unstructured Data weight more than Structured Data for emotional purchase.

Business Turmoil

In a business turmoil, it is easy to understand that seasoned manager experience will deliver better business forecast than Structured Data from a junior planner. It will be difficult - if not impossible - to structure with a mathematical model the experience of the seasoned manager.

Up-Stream or Down-Stream of the E2E Supply Chain

Based on our position in the End-2-End (E2E) Supply Chain of a product toward the consumer, we could think that more our business is up-stream (consolidation exist and we are far from consumer emotional purchase act) more Structured Data will provide an accurate forecast. But then why the Wipe Bull effect exists?

Here come in play the emotional decision (or Purchase Act) of all stakeholders in our End-2-End Supply Chain flow. But there is may be a solution: increase data visibility along the E2E Supply Chain. A clear backward information flow (see below) should be integrated. This will be possible only with a Win-Win attitude between all partners of the supply chain. 

May be a Kyoryoku Kai (Supplier Association) from all Supply Chain stakeholders should be implemented. The Japanese Automative Industry (and Toyota in particular) taught us how to move forward. To start with forecast transparency, VMI (Vendor Managed Inventory) could be a solution in E2E Supply Chain. Forecast accuracy can gain easily 20% and OTIF reach some records above 98%.

Big Data

How will our forecast process evolve with the « Big Data » paradigm change that we are in? Will transparency associated with Big Data promote Structured Data ? Or will the combination of Structured and Unstructured Data remain the best solution ?

The answer will be open for few more years. It will also impact planners job and their value proposition.

Integration of Structured and Unstructured Data

The integration of Structured and Unstructured Data (including Gut Feeling of C-level leaders) is the best solution to pursue. The characteristic and the maturity level of our industry, company and product will determine what weight we should give to each information flow. Demand Review, S&OP and IBP are surely the right communication and decision platform for this integration. S&OP and IBP will help us to transition information from "Gut Feeling" and/or "Unstructured Data" to "Structured Data". This information will become mindful for day-to-day operations. The engagement of all stakeholders is key in the S&OP/IBP process. But this will be part of another discussion. 

Please let us know your comment on how you weight and  how you integrate Structured and Unstructured Data in your day-today and S&OP work?