AI in Practice

Seems like yesterday when we all became aware of ChatGPT. My awakening came from the LinkedIn platform. It was where I first noticed a post or two generated by the technology with each post featuring a statement at the end along the lines of “and, this post was automatically generated by ChatGPT”. That was November 2022 and is where I demark the start of Artificial Intelligence (AI) as we know it today. 

However, the reality is that what people now generally refer to as “AI” had its start long before ChatGPT.

The current crop of generative AI tools (Claude by Anthropic, ChatGPT by OpenAI, Gemini by Google, etc.) had their start from a seminal paper published by Google in 2017 called “Attention is all you Need”.  The paper outlined a new ‘network’ (read: algorithm) approach, the “Transformer” which eschewed recurrent or convolutional neural network approaches for language translation in favour of an approach/model that required significantly less time to train, was parallelisable and required less GPU energy/processing time. The Transformer research is the basis of all the GenAI tools we’re familiar with today. And that is only the Transformer Research which goes back to 2017; the reality is that what we now call “AI”, including concepts such as predictive analytics, machine learning and many of the common algorithmic models, goes back for decades. This broader definition of AI and associated applications is where AI gets interesting and where companies can generate value.

In practice, businesses need to consider this broader application of the technology, beyond the simple chatbots that answer questions and summarise board papers. This is an area I’ve been focused on for over a year now. My experience has generated several practical learnings on how to learn about and adopt “AI” in a fashion that delivers actual business benefit. 

This post outlines the approach and key learnings.

Start With Principles and a Common Language

For me, the company journey with AI started as a simple request from the board in early 2023. The request was “what are we doing to manage the risks of ChatGPT in our business”? The popularised concern at that point was related to what information people were putting in into public models such as ChatGPT, what controls did we have in place to monitor what was being shared and how do we gain comfort that we’re monitoring emerging risks resulting from the new technology?  As would be expected, the request forced me to think critically about AI deployment and resulted in a board paper outlining the steps we were contemplating to manage risk. 

In our case, we took three immediate actions. First, we implemented some AI access controls using a technology we had previously deployed.  The technology already in place, called Digital Loss Prevention (DLP), is used to monitor, control and prohibit sensitive information from leaving the organisation. There are many tools available for this and being a Microsoft shop, we used ‘Purview’, a Microsoft-based Digital Loss Prevention (DLP) software.  Tools such as Purview use deep content analysis to detect sensitive data in files, emails, and messages, and then enforces actions such as blocking sharing, encrypting content, or providing real-time guidance to users. We pointed our DLP solution to the Internet URL for ChatGPT and configured rules to block certain types of information from being uploaded to the GenAI tool. It was an important first step for us in our AI journey.

The second step was to define basic principles by which to guide our usage of AI tools, such as ChatGPT, and also other categories of ‘AI’. Initially, the policies manifest as an email from me, as the CIO, out to the business with basic definitions and guidance on how to best use the ChatGPT. The principles were basic in that they advised our staff that public models may train on whatever information is shared and requested that users consider the implications prior to sharing any confidential or proprietary information with the tool.  The email also provided guidance as to where to seek additional information if they had questions. Essentially, it was an acknowledgement that it was ok to experiment with the new technology but to do so in a responsible and ethical fashion.

Lastly, we embarked on an education campaign. First with the board, in the form of a more fulsome paper on the opportunities, benefits and risks of the emerging technology and also with the broader business on what AI is (and what AI is not!) and how it can help with productivity and efficiency. A similar exercise was done with my peers on the executive team so that they could carry key messages through the various lines of business they managed.  Once completed, we had an awareness, a base-level of knowledge of the capabilities and consistency in language regarding AI.

Experimentation

After the initial comms and as the ‘AI’ craze gained momentum through 2023, we embarked on some organised initiatives to put the tools in the hands of the business.  This included business workshops to further educate on the technology and discuss the different types of ‘AI’ available.

Through these workshops, we solicited business specific ‘use cases’ to experiment with the technology.  All up, we identified 80+ different uses cases which we organised through an affinity mapping exercise into categories including member/customer experience improvement, growth/marketing applications (cross-sell/up-sell) and efficiency/cost savings opportunities. The use cases were then prioritised by business impact vs. ease of implementation and served as the starting point for our AI deployment strategy.

We also created a simple ‘reference model’ to use with the executive, the board and the business more generally. Using this model, we created a common language and understanding around AI and its uses. 

At its core, the reference model outlined three general categories of AI utilising terminology borrowed from Gartner; (1) Self-service AI, (2) Custom AI, and (3) Embedded AI.

(1)    Self-Service AI – this is an AI category that an individual would typically use on their own to help in everyday tasks. ChatGPT, Claude, Perplexity, etc. are all examples of what we called ‘Self-Service AI’;

(2)    Custom AI – this is an custom trained, self-contained Large-Language Model (LLM) trained for a specific functional purpose in the business;

(3)    Embedded AI – this is AI that comes with or is embedded in an existing software solution. Most software providers such as SalesForce or Microsoft are embedding AI into their existing software offerings.

Reviewing the use cases we identified during the initial workshops along with the reference model, we selected one use case for each of the three categories of AI (Self-service, Custom, Embedded) to pilot with a test & learn mindset.

Self-Service AI Pilot

We initially procured Claude, Anthropic’s generative AI solution (professional edition) for our executive team. Before long, it was not uncommon to hear someone in an executive team meeting say “Claude says we should do this/that”.  Soon, we had the senior leadership team (direct reports to the executive) clamouring for Claude as well. We then deployed Claude (enterprise edition) as a Self-service AI tool to approximately 80 people across the organisation. Along with the deployment, we provided a structured program of education, training and interventions to ensure adoption across the cohort.

Embedded AI Pilot

Our embedded AI pilot used an existing knowledge management tool previously deployed in our environment. This tool is essentially a knowledge management tool used by front-line personnel to support member queries. We simply enabled the AI ‘feature’ in the tool which then trained the tool on our existing knowledge base. The benefits were immediate. Examples of the benefits included a reduction in induction training for call specialists, a general increase in knowledge usage across the business and most certainly an indirect increase in member interaction/satisfaction through summarised information more quickly.

Custom AI Pilot

This was the most complex and comprehensive pilot of the three. The Proof-of-Concept (PoC) we selected related to call quality analysis. We took converted 15,000+ calls to text and anonymised the information (to protect Personally Identifiable Information). We then trained an OpenAI custom model using off-the-shelf Microsoft tools such as Azure Transcribe, Azure PII, Azure OpenAI GPT4o and Azure OpenAI Embeddings. In addition to the raw text from the calls, we ingested other data sources into the model for training purposes including our internal call quality metrics/principles, our communication style guides and Private Health Insurance (PHI) legislation (example – Private Health Insurance Act 2007). 

Once trained and grounded (AI term for the process of connecting the model output to specific, verifiable source of information to improve accuracy), we again used a standard Microsoft visualisation tool called “PowerBI” to create dashboards for the call quality team to monitor. The results were astonishing; we went from 2-3% call coverage via manual listening to 100% call coverage using the model with increased accuracy around call reason codes and call sentiment analysis. With the original quality process around inbound calls involving a small team of individuals who listened to calls and manually (semi-subjectively) evaluated the quality, one of the learnings was that we need to re-think the overall process based on our learnings and rich volume of the insights the pilot surfaced.

Outcome and Next Steps

Each of the pilots was successful in that each offered rich learnings for the category of AI. Our intent was to experiment across both the use cases but also the ‘complexity’ of the use case which provided another dimension to the learnings.

An interesting and opportunistic element of one of the pilots was a partnership we formed with the University of New South Wales (UNSW).  Through an employee who was studying AI for his degree at UNSW, we supported him on his research project in investigating the overall effectiveness of different ‘interventions’ as our people were learning to use the technology. The interventions took different forms including targeted training, specific guidance and/or general support for those struggling to take full advantage of the AI technology.   

In terms of next steps now that we’ve consolidated the initial learnings from the pilot(s), we’re planning on extending the ‘Custom AI’ pilot to other business units and adding auto-generated call notes and real-time sentiment analysis for calls. The other focus with learnings in hand, is to develop a Responsible AI Framework (RAI) to guide future direction as we continue to deploy. 

It sounds rather fancy, an RAI, but essentially the framework is comprised of a formal policy statement which integrates with our existing Risk Management Policy framework, a taxonomy of potential risks based on our specific use cases and most importantly, a mapping of the risks to our existing controls framework to ensure coverage across security operations, data governance and Line 1 risk management.

Practical Learnings

It’s difficult to summarise all the learnings over the last 18 months into a concise bullet point list.  The richness of the information across the multiple pilots, the feedback from the users as they adopt & adapt as well as the governance learnings is overwhelming.  That said, here a few key learnings that rise to the top of the heap.

It Starts with the Business

Make no mistake about it – Artificial Intelligence equals business change. My experience as well as the experience of others I’ve spoken with started with AI as a technology ‘thing’. This is not a technology thing. Avoid that pitfall.  Sure, AI requires a level of engineering and technical expertise which is necessary, but not sufficient, for success. Our Custom AI call quality pilot was led by the business with IT playing a background role in facilitation & coordination. Focus on the business value first, then think about how AI that can drive benefits and force the business to lead.

Avoid Naval Gazing

One of the best things we did is quickly assess the available technologies and then make a firm decision as to the direction we wanted to head.  As mentioned previously, we’re a Microsoft shop so the Azure stack with OpenAI made a lot of sense for us. With Claude (our Self-service option), we chose that over co-pilot because several of my peers at the exec were already familiar with Claude and its interface. At the end of the day, it’s more important to get started and learn than overanalyse the tooling. Make a reasoned decision on the technologies, then get started and don’t look back. 

Plan for Downstream Impacts

One of the assumptions among the management team was that once deployed, an AI solution needs little additional funding and should provide immediate cost & efficiency benefits. This is simply not true and in fact, our experience was that we needed additional resource in the short term as we got up to speed with the technology offering. A key learning for us was in regard to the call quality pilot. The LLM/tool suggested we had 10x the number of complaints than we currently thought and we worried that we may have a potential compliance issue. Fortunately, this was not the case however we did need to build more robust downstream processes to manage the increased throughput the solution provided. Plan for a surge in the initial cost and resource requirements before you can begin to bank the savings.

Tone at the Top’ is Paramount

Our executive team and in particular, our Managing Director were massive supporters of AI. Whether in informal meetings or formal ‘Town Halls’, the message around AI was always one of opportunity and excitement. It became contagious. By first deploying to the executive, we created a bit of FOMO around the technology which spread quickly. We were fortunate in that our board, once educated, was also very supportive of the controlled experimentation approach.  Take the time to get alignment and full support among the Exec and board before launching business initiatives.

Organise your Data & Infrastructure

An old saying in computers is “garbage in, garbage out”.  This has never been truer than in the realm of AI. Whether training a custom LLM or simply having a DLP solution in place, having inventoried and clean data whose lineage is well controlled and understood is paramount to effective AI. Fortunately for us, we had embarked on a serious refresh of our data infrastructure and invested heavily in bolstering our maturity around data governance prior to starting the AI journey. These investments came to the fore quickly as we started our proof-of-concept work.  Having your data sources catalogued, data owners assigned and an overall process around data governance will only expedite your AI journey. Invest in data governance early and recognise the foundational role data plays in the training & optimisation of AI models.  

Conclusion

AI is an general term which represents different categories of algorithms, machine learning and applications. Given that AI is an umbrella term, using ‘AI’ could mean many things to many people. Like the concept of ‘drugs’, context around AI usage is very important.

Using drugs ‘prescribed’ by a doctor is different than experimenting with drugs with your friends. Using a pharmacy to source your drug(s) is different than getting drugs from the corner dealer. Drug use, whether prescribed or recreational, is best done in moderation with constant monitoring of symptoms/side effects of usage being a suggested best practice.

Using AI is the same; read the label, only use when directed and don’t exceed recommended dosages!

(Note: no Generative AI was used/harmed in the writing of this blog post :-)

Jeffery Eberwein is a senior business executive specialising in technology, data and artificial intelligence and their implications for business. The views expressed in this article are the views of the author and not the views of any affiliated or referenced organisation, either directly or indirectly.