Why Context in Analysis Matters

I’ve been around the block a time or two. I sometimes feel like the poster boy for the graying workforce. Maybe it’s an upside and maybe it’s a downside but as I listen to the discussions around big data, the Internet of Things and the resurgence in analytics, it seems like I’ve seen this movie before. In the 80’s I was working with manufacturing companies to help them adopt SPC, SQC, Design of Experiments and Exploratory Data Analysis.

I watched companies struggle to find the skills necessary to drive adoption of data analysis. Do we hire statisticians and teach them about the process? Do we teach statistics to process engineers? Now the conversation is about ‘data scientists’. It seems pretty familiar.

I work in an industry that has employed distributed networks of sensors and collected minute-by-minute readings from literally thousands of instruments for decades. Isn’t that the ‘Internet of Things’? (Read an interesting post on this topic from Peter Reynolds at ARC).

We still struggle with the precursor to analysis – preparing the data. We even have a special term for it - data munging. It consumes a significant fraction of the time allotted for analysis. I spend a lot of time talking with customers about their problems, challenges and goals. They complain of the time, effort and skill required to complete ad hoc analyses when it involves reaching into multiple systems to find, extract, merge, align and condition data for analysis.

We’ve long since passed the point where analysis of process readings is sufficient to resolve production issues. Many problems require data from the process data systems, LIMS systems, maintenance systems and ERP systems. Critical information is still locked up in paper logbooks or worse still, wiped off of the whiteboards in the production briefing area at the end of each shift. That sort of unstructured information can be the difference between success and failure in resolving process and production problems.

I believe context will be especially important as businesses rush to adopt predictive and prescriptive analytics. The ability to improve pattern classification is lurking in the context of the data. What actions should we prescribe for a given process upset? Part of the answer is in the context of the data. What was done in previous similar events? What worked? What didn’t?  

Context is the story behind the data. Read more…