Story of the missing data...

Are we challenging our own assumptions? Are there any critical insights we are missing because we are not looking beyond what's visible to us? 

There is so much to learn from the stories told, projects completed, and blockbuster companies launched. But there is also a lot to learn from the stories that were never told, the projects never completed, those who didn't survive the battlefield, and the companies that failed. Survivorship is only half the story, and it is a bias that hinders how we make our judgement. There is power in the data and storytelling but if we only looked at what's in front of us, we are significantly skewed in our interpretation and led astray in our search for the truth. 

Survivorship is only half the story, and it is a bias that hinders how we make our judgement. There is power in the data and storytelling but if we only looked at what's in front of us, we are significantly skewed in our interpretation and led astray in our search for the truth. 

The most famous example of survivorship bias dates to World War II when the US military asked the Statistical Research Group at Columbia University to examine the damage done to the planes that had returned from the missions. The team focused on the bullet holes that caused extreme damage in the areas to determine where armor could be added to increase the protection from the flak and bullets. After careful evaluation of the damaged areas, it was decided that the additional armor should be added to the areas of the planes that showed the most common patterns of damage. A statistician Abraham Wald working on the project challenged this conclusion. He encouraged the team to focus on not what was in front of them, but on to something that was conspicuous by its absence, "the missing holes." He pointed out that the planes being examined were only those that survived, therefore the study was missing a critical set of datasets "the planes that didn't make it back." 

This theory which later became known as "Survivorship Bias" acknowledges the fallacy of judgement driven by concentrating on the people or things that made it past some selection process failing to consider those that did not. It can be attributed to the fundamental misunderstanding of cause and effect, and a misunderstanding of correlation versus causation. The bullet holes that the military was examining closely and recommending remediation indicated the areas of the plane that survived the bullet – they were much stronger and not the area that needed reinforcement.

We are increasingly becoming sophisticated about our data analysis process, but what's critical is “not just the interpretation and analysis of the given data, but also of the data that we cannot see, that was not given to us.” While dropout turned entrepreneurs make a fascinating story, there are plenty of dropouts who ended up not making any cover page story. Selling those success stories as courageous, unconventional, and innovative create logical fallacies for many, especially young minds. Sendhil Mullainathan, a professor of computation and behavioral science at the University of Chicago Booth School of Business talks about survivorship bias and the poor decisions it can produce in a podcast Choiceology.

We are increasingly becoming sophisticated about our data analysis process, but what's critical is "not just the interpretation and analysis of the given data, but also of the data that we cannot see, that was not given to us."

While survival stories are impressive to read, that's not the full story. By analyzing the projects, products, companies that never made the headlines, we develop intelligence to understand the inherent and material weaknesses that need remediation. While great resignation wave is real based on the massive data and media frenzy, there is also a flip side where many have stayed back and continued to drive the company forward. But because all the visibility and attention shifted to talent gap and hire fast and at the asking price, those who stayed back felt unrewarded, eventually taking that exit door too. End result, companies hired many but also lost many and the talent shortage continued. With So Many People Quitting, Don’t Overlook Those Who Stay (HBR Article) Additionally, are we taking the time to carefully investigate if the hiring frenzy is driven by the real organic growth or is it because the process is so broken that it takes 50 instead of 5 resources to solve the problem? Just adding resources to address a given challenge is not the right solution, it will eventually spiral out of balance. Look for the real issue, correlation is not causation. For real progress, we need to fix what's broken.

Don't use the data to just support your hypotheses. Be sure to look for the missing holes... 

#Dataanalysis #Storytelling #data