Don’t wait until it’s clean

“I’m not going to introduce BI until I have got all the data cleaned first.”

I hear this all the time from all sorts of organisations. I kind of get why people would think this way, you don’t want to deliver reports that aren’t correct, this would lead to mistrust and a lack of adoption.

Anyone delivering a BI project want to ensure that they are delivering value to the end users and accuracy is at the forefront of adding value.

For me though, BI can make a huge impact in the process of cleaning data. I have seen almost every organisation I have worked for quickly find inaccuracies or errors in their data as soon as they start visualising it.

One example I have heard out of Australia was in relation to a company that measured it’s Qantas expenses. As far as they were concerned it was about $175,000 per year, when they put their data into a BI tool, they very quickly identified a series of misspellings : Quantas, Quntas.. you get the point.

They could have crawled through the data and tried to ‘clean’ it first, but with a BI tool they were able to find it and identify all of the specific examples quickly, allowing them to go back to each individual record and fix them without touching any other records.

Another one I saw recently was in a health organisation that measured average times for their practitioners on specific treatments. For some reason their average time was higher than they expected. Once we created a couple of charts and a table we quickly identified one event that had been entered as 599 instead of 5.99. Being able to identify this so quickly meant that they could fix it before any of the charts were deployed to the organisation.

The bottom line for me is, if you are in a data cleaning process, then a BI tool can act as an enabler, not something that you wait for. It can make the job of profiling your data easier, identifying and allowing you to fix errors much faster.