Chat-bots - A worked example : Why does the first line support role exist in these times of AI being so accessible in the cloud?

Following up from a conversation with an old friend and ex-colleague earlier in the week I was prompted to walk-the-walk and see how difficult it really is to set up an AI Natural Language Processing (NLP) tool in the cloud for first line catch and dispatch / support across either front line support or specialist knowledge domains such as policies.

Turns out not very and after two hours I had a Q&A chatbot up and running on my chosen riveting topic of regulatory reporting policies using the AWS Kendra tool.

Anything AI / Machine Learning / Natural Language processing carries all the baggage of technical mystery and obfuscation. Combine that with cloud based and the wizardry is squared, the scepticism cubed…but this proves deep understanding of how these technologies work is not required and a strong monitoring and feedback framework will give far more insight, control and business value than can be achieved with manual processes alone.

To de-mystify how this can be done, at the very least prove the art of the possible, I have documented the steps below in the style of a school science experiment and the Scientific Method. This is to celebrate my days as a home tutor being firmly over and my daughter assures me it is a mutually held feeling.

(Spoiler – it does work and for those speed reading through, skip to Conclusion and Next Steps)

Aim

Prove whether AWS Kendra can do effective first line interpretation with its Natural Language Processing and provide accurate results to free text questions.

Apparatus

AWS Kendra – provide the natural language processing and free text search / matching capabilities

AWS S3 – Simple Storage Service - Storage bucket for the raw policy documentation PDF’s and Q&A source

Test policy documents - Financial Services regulatory documents of Fundamental Review Of The Trading Book from European Banking Authority, Basel III : Finalising post-crisis reforms from Bank For International Settlements, Benchmark Reform and Transition from LIBOR from ISDA.

Test Q&A file : Excel CSV formatted file with some expected questions, answers and links

Method

Step 1 : In the AWS Console I first selected the ‘EU (Ireland)’ region, a region that supports AWS Kendra

Step 2 : In the S3 console, click ‘Create Bucket’ to create two document stores – chatbottest-source-docs and chatbottest-source-faq

Step 3: Click on the buckets to upload the documents you want to be part of the FAQ

Step 4 : Do the same for a sample FAQ, as per below, and upload into the FAQ bucket

Step 5 : Go to the AWS Kendra service (select matching region Europe in my example) and click ‘Create Index’

Step 6 : Name your index, here I chose ‘ChatBot-FinanceRegulations’ and, under IAM role, select ‘Create a new role’. Leave all other options for this example.

Step 7 : This is a moderately technical part, you need allow AWS Kendra to see the S3 buckets created in the first steps. A sledgehammer approach is to give the new Kendra IAM role read only access to all of your S3 buckets, OK for a test only.

To do this go to the AWS IAM (Identiy Access Manager) console and select Roles :-

Select the AmazonKendra newly created role and click ‘Attach Permissions’. Filter the list by entering S3 and select ‘AmazonS3ReadOnlyAccess’ to enable, it should look like this once done :-

Step 8 : Go back to the AWS Kendra console and select your next Index, then select ‘Add Data Sources’, then select the Amazon S3 ‘Add Connector’ button. There are a selection of other sources available but I'm keeping it simple with S3 here.

Step 9 : Give your data source a name, I chose ‘ChatBot-FinanceRegulations’ and select Next.

Step 10 : Click Browse S3 and choose you documents directory.

Step 11 : Select the AWS Kendra IAM role to match the one create earlier

Step 12 : Choose a ‘Set sync run schedule’ of ‘Run On Demand’ and then ‘Next’

Step 13 : Quick review of your settings and click ‘Create’ to create the index

Step 14 : Once successfully create a green banner will appear at the top of the screen, click the ‘Sync Now’ button to fill the index. 

Step 15 : Go to your newly created index by selecting ‘Indexes’ on the left and then the name of your index, you should see something like this :-

Step 16 : Follow the same instructions to add the frequently asked questions by selecting ‘FAQ’s’ on the left (the above example I had already loaded these)

Done! The chatbot index is now all complete and ready for use by select ‘Search Console’ on the left.

Results

This is just a subset for brevity and to show the different capabilities.

Test 1)     Trying a business specific term and the search capability over the policy documents – ‘how do i calculate loss given default’ gives the following results :-

Successful start given no training has yet been done on the database. Links and extracts provided into each of the documents and multiple options for people to look for. The thumbs up / thumbs down icon provides training for future answers.

Test 2)     Check the FAQ with a simple question of ‘What does EAD mean?’

Success again but the FAQ source needs to be higher priority here – the link to the definition specified in our FAQ is displayed and we also have the usage in the policy.

Test 3) Test the natural language capabilities with the same question different ways – ‘what does EAD stand for?’, ‘EAD what does it mean?’, ‘how do i find out what ead means?’

All coming back with the correct answer, either as per test 2 results, or the results above.

Conclusion

It was proved that AWS Kendra can effectively interpret documents and natural language questions providing coherent answers which can then be trained and improved.

The role of the knowledge management function now needs be curating the index, monitoring of the accuracy of the responses, and constantly tuning the results. This pivot allows pro-active knowledge management and insight to both the questions being asked.

Additionally the response to questions is immediate so no wait-time for people asking questions, savings on effort managing tickets and, working on the basis that for every person asking a question ten people are thinking it, a significant organisational capability improvement.

The list price-point of $5k a month (before any corporate discounts) makes the effort saving case a compelling story, but you do need to factor in the knowledge management and referral handling costs on top of this rather than a straight comparison to existing costs.

At $1.8k a month for the developer edition to test out the capabilities it gets more compelling to run an agile proof of concept. Remember, this is in the cloud so no test hardware to procure or delay the process, but make sure the data security guardrails are in place if non-public data is involved.

I am convinced this approach, combined with process and data, is going to be core tooling for the Chief Knowledge Officer role of the future.

Next Steps

This has proved the foundational building block so my focus has moved to how this results in a successful business outcome :-

i)     Defining the business process life-cycle and control model this needs to sit within and tools required

This approach of Cloud Natural Language Chat-Bot only works within an effectively curated knowledge base. Continuous sampling is needed of the questions asked, referrals where users select that their question have not been asked need prompt follow ups, and the Q&A knowledge base itself needs control and a process to manage changes.

ii)     User interaction

A simple web user interface to interact with the users, display the ranked results from AWS Kendra, and get feedback on whether their question was answered. Whilst the default interface worked perfectly well for this test, more control of the workflow is needed and understanding of the interaction patterns. Embedding with in an organisational context would be key.

iii)     How the feedback is integrated into the continuous improvement of the organisation.

Answering the questions of the organisation in a quick and cost effective way is still only a remedy rather than curing the underlying problem thereby reducing organisational friction. A high performing organisation should be able to take these question signals and diagnose why they were raised in the first place,  pro-actively eliminate the underlying issue through improved training, feeding into Agile development backlogs, making the user experience more intuitive or reworking a process.

iv)     Real-time and moving data

Connecting AWS Kendra  to large data sources / ticking data to see if the same level of coherence is achieved and explore how the knowledge management process evolves with moving data.

To Finish

I hope this has been of interest and piqued some curiosity. Even better if you give it a go, I tested the above on the AWS free tier so nothing stands in the way of some free-style innovation.

Feel free to contact me if you would like any further information or to discuss.

Phil Kent, September 2020