Is Business Intelligence outdated?

in the age of Ai and Machine Learning, is it still worth to teach & learn about and discuss Business Intelligence?

As I was preparing to start teaching my Business Intelligence (BI) classes (undergraduate and graduate) at the Ageno School of Business at Golden Gate University , I was contemplating how my course fits into 2024, an environment where everybody talks about AI and ML. Is what I'm teaching outdated? Is BI even still relevant in the age of AI, and are Data Warehouses and their technology outdated?

After reflection, the answer to these questions is in my opinion a resounding NO, BI is still very relevant and needed, and so is the technology developed for DWHs.

BI is about using data to understand and remediate business problems. Operational systems like the ERP (supply chain, HR, Payroll, Accounting) are good at collecting and structuring data for a defined purpose. They are built to support business processes, and in the process of conducting business processes, data is generated, processed, and stored.

But in order to remain competitive, it is necessary to constantly review and improve business processes. BI has three core areas - descriptive analysis about what has happened in the past, predictive analytics about what will happen, and prescriptive analytics about what should happen.

Descriptive analytics are common, yet still underutilized today. Let's take the recent example of the Boeing Max 9 incident - you might recall that a brand new jet that just entered service in October 2023 on a flight from Portland, OR in early January 2024 lost a door plug. Subsequently airlines that have many Max 9 jets found loose bolts. Without access to the ongoing FDA regulation, how could analytics uncover such problems? For each manufacturing step there is supposed to be a benchmark - required steps and the time it takes to complete the steps. In the case of mounting the door plug, there are steps (inserting the pre-assembled door plug, inserting bolts, torquing the bolts to spec, etc.) that need to be completed and take a certain amount of time. Do they measure the steps and how much time they take? You have to assume so, because it is required for your project plan. There are probably 1000's of processes required to assemble an airplane, each with predecessors and follow up (such as quality control), that each take a allotment of time. Measuring the actual time it took compared to the benchmark is critical - if it takes too much time, production targets are not met. Is there a specific process that consistently has overruns? What are the causes of these time overrun - maybe the benchmark has to be adjusted, maybe training or tooling has to be improved, maybe a problem in the supply chain has to be fixed. A dashboard would show the status of all process and allow managers to zoom in on the ones that quire attention. But what if a process consistently faster than the benchmark? What are the reasons for that - could it be that corners are cut or quality assurance is skipped? Usually we are focused on delays, and time savings are welcomed and celebrated. But in fact any deviation from the benchmark should be analyzed.

Bottom line - there are really important insights that descriptive analytics can deliver, and they could unveil a root cause issue, for example lack of quality assurance. 

Ai is not going to replace this data analysis work. AI requires that data is collected and cleansed - just as we do for a data warehouse. The collected and cleansed data is then used to train an algorithm for a repetitive task - that is what machine learning is. There was initially a school of thought that you could just throw any data into a data lake and let the machines figure out what data is important. But that didn't lead to insights, only to data swamps with lots of data that was not really utilized. Nowadays people talk about Lake house concepts, which combine the versatility of data formats from a data lake with the data governance of a data warehouse - evidence that the concepts of the DWH-era are not obsolete and still very much needed.

ML can be used to capture more data and make more data accessible for analysis, for example through natural language processing. But the data still has to be analyzed, and this analysis process is not highly repetitive, so it can't be replaced by ML.

How about AI? Could you ask ChatGPT "why did the bolts one the Boeing Max9 door plug fail" and ChatGPT, being the large language model that it is, would mine its databases (including the internet), find if somebody has ever written reports about failing bolts in the Max9 door plug, and then summarize all the data it finds - because that is what generative AI does, unless it hallucinates and just makes stuff up. Very likely, however, generative AI will draw a blank in this case. Because there are thousands of parts in an airplane, of which few ever fail, and even fewer on a relatively new model, is is probably not much a generative AI model can feast on. So it is highly unlikely ChatGPT or other large-language model based AI will provide much insight into the root cause of this problem. But believe it or not, good "old fashioned" BI with descriptive analysis might actually offer a clue if indeed process times for door plug installation and quality assurance have been below benchmark times (maybe in an effort to make up for production delay in other areas, since the door plug installation comes towards the end of the entire assembly process?)

So in conclusion - yes, Business Intelligence is very much needed today in the age of machine learning and AI, because gaining insight from data is an exploratory process that requires some business acumen (in order to ask the right questions) and some technical skills (to bring the right data together). For the latter part, we have now better and more modern tools available, and maybe even AI support in the creation of data visualization. But the analytical process itself requires YOU - a human trained in data analysis who applies their knowledge and experience to allow heretofore unknown insights, rather than highly repetitive work. 

So in my class we will still discuss how to extract and cleanse data from operational systems, load them into an analytical data store (which could be a Data Mart, Data Warehouse, Data Lake, Data Lakehouse or whatever you want to call it) and then apply analytical methods to gain insights from the data through data visualization such as dashboards. And I really believe we all need to do more of this because we just can't rely on AI to do it for us.