Deploying Agents for Knowledge Extraction from Supervised Models Using LLMs

In a previous article, I had detailed how I used GPT 3.5 to create understandable narrative of each customers churn probability for call center agents. [Shap LLM for Call Center Assistance]. In short the objective was to create an explanation for each customers churn probability prediction and what could be potential actions to reduce churn. I wanted to try local LLMs with open source models like Llama2, Gemma 2B, Mistral etc for the same. Happy coincidence Llama 3 released last week too so was able to try that out too.

Updates made to Architecture:

1. Created a Shap feature dependence document from SHAP PDP in text format and added it into RAG context documents. This file will provide details of each features contribution and distribution towards target. Replaced decision tree file with this.
2. Created a vector database with ChromaDB. Embeddings for RAG was created using few different baseline open source models [Finally selected all-distillroberta-v1] and saved to ChromaDB collections locally
3. All the model process is done with Ollama locally. It makes it very easy to load models locally and have good compatibilility with langchain and chromaDB.

Results from Call Center Assitance Task:

The same prompt used with GPT 3.5 was passed to Llama3 model. The results were much better in quality. Here is a sample result provided on an individual customer

The customer, a male with dependents, has internet service through Fiber optic. He uses online backup, device protection, and tech support, but doesn't use streaming movies. His contract is month-to-month, and he pays electronically. He's been a customer for 22 months.

Predicted Churn Status:

The predicted churn status is "Very likely" with a probability of 0.69.

Top 3 Reasons for Potential Churn:

1. Streaming movies (No): The lack of streaming movie service increases the likelihood of churn by 1.80%. This suggests that the customer might be dissatisfied with the current entertainment options, making them more likely to switch providers.

2. Contract type (Month-to-month): Having a month-to-month contract increases the likelihood of churn by 16.37%. This could indicate that the customer is not satisfied with their current service and is more likely to explore other options.

3. Monthly charges ($89.4): The monthly charge of $89.4 increases the likelihood of churn by 10.99%. This might suggest that the customer feels they're paying too much for their services, leading them to consider switching providers.

Actions to Reduce Churn Likelihood:

1. Offer streaming movie options: By offering the customer streaming movie options, we can reduce the likelihood of churn by an estimated 1.80% (based on SHAP contribution). This action helps because it addresses a specific dissatisfaction with the current service.

2. Switch to a more favorable contract type: We could switch the customer's contract type to a year-long or two-year agreement, which would decrease the likelihood of churn by an estimated 16.37% (based on SHAP contribution). This action helps because it reduces the customer's uncertainty and commitment issues.

3. Discount or adjust monthly charges: By adjusting the customer's monthly charge to a more competitive rate, we can reduce the likelihood of churn by an estimated 10.99% (based on SHAP contribution). This action helps because it addresses the customer's perceived value for their services.        

• This looks very good and detailed.
• However it takes around 1.5 mins to create the whole prompt repsonse with any of the local LLMs. I am running it on a system with 6G of GPU. With a more powerful system it could come down to a lower time. Comapre this to Open AI GPTs which was able to run the prompt in ~30 seconds although at a cost

Creating Agents for Knowledge Extraction:

Often times whenever I had to deploy a model, I used to have multiple iterations of runs, long cells of plots to understand the feature patterns towards target. With tree based models like XGBOOST, SHAP is always the tool to look at to explain the model patterns. This helps me understand the trends and patterns of the model as well as able to explain some insights to the end user. Although there are few challenges that needs a solution here:

1. This is a time consuming process. One have to sit and analyze the patterns from shap summary plots and partial dependence plots. This is also a little bit technical. Then you have to document with details of patterns you have found from manual analysis
2. When models goes to production, we usually track the performance metrics of models only. Feature patterns towards target is not analyzed on every update. This might create an issue when actions are done on past assumptions shared from SHAP analysis at development which might be outdated.
3. Extend of manual intervention needed to identify best actions to recommend to optimize the target

Having an agent who would accurately extract the feature patterns and provide actionable recommendations would solve this problem. This can be easily automated and updated with every production update.

After using local LLMs it came to me that it is quite easy to create an agent system with defined roles. So I decided to create a Business Analyst - Manager agent system on the same dataset (Telco Churn). I created an agent system as detailed below :

Basic idea here is to create 2 agents in a sequential manner.

1. The first agent is tasked with a business analyst role and is asked with queries that needs to be answered. This agent built on Llama3 basically uses the same ChromaDB for getting the context embeddings and returns the report
2. The second agent is tasked with a managers role. This agent gets the same queries as business analyst. However his/her task is to learn and adapt from the business analyst agents response. The context here would be the response from business analyst agent

Here is the response from the business analyst agent for the query passed:

 **Report: Main Reasons for Churn and Recommended Actions**

As a business analyst for a telecom company, our goal is to identify the main reasons for customer churn and recommend actionable steps to prevent it. After analyzing the available data and features, we have compiled a comprehensive report highlighting the key findings.

**Main Reasons for Churn:**

1. **Contract Term:** The type of contract term has a significant impact on the probability of churn. Specifically:
	* Month-to-month contracts increase the probability of churn by 16.37%.
	* One-year contracts decrease the probability of churn by 9.98%.
	* Two-year contracts decrease the probability of churn by 19.89%.
2. **Payment Method:** The payment method used by customers also affects their likelihood of churning:
	* Credit card (automatic) payments decrease the probability of churn by 2.94%.
	* Electronic check payments increase the probability of churn by 4.30%.
3. **Paperless Billing:** The use of paperless billing is another factor contributing to customer churn:
	* Paperless billing decreases the probability of churn by 3.80% when it's No, and increases the probability of churn by 1.67% when it's Yes.
4. **Streaming Services:** The availability of streaming services also plays a role in customer churn:
	* Streaming TV (Yes) increases the probability of churn by 1.61%.
	* Streaming Movies (Yes) increases the probability of churn by 1.80%.

**Top 5 Recommended Actions to Prevent Churn:**

Based on our analysis, we recommend the following top 5 actions to prevent customer churn:

1. **Pitch Two-Year Contracts for Month-to-Month Customers:** Since month-to-month contracts increase the probability of churn, pitching two-year contracts to these customers could decrease their likelihood of churning by 19.89%.
2. **Optimize Payment Method for High-Risk Customers:** Analyze customers who are at high risk of churning and adjust their payment methods accordingly. For example, if a customer is likely to churn due to using electronic checks (which increases the probability of churn by 4.30%), consider offering alternative payment methods like credit card (automatic) payments.
3. **Implement Paperless Billing for Low-Risk Customers:** For customers who are less likely to churn (i.e., those with two-year contracts), implement paperless billing to decrease their likelihood of churning by 1.67%.
4. **Offer Streaming Services to High-Value Customers:** Identify high-value customers and offer them additional streaming services, such as streaming TV or movies. This could increase the probability of retention by 1.61% for streaming TV and 1.80% for streaming movies.
5. **Monitor and Adjust Contract Terms:** Continuously monitor customer contract terms and adjust them accordingly to minimize the impact of month-to-month contracts on churn.

**Justification:**

These recommended actions are justified based on the analysis of the available data and features. By understanding the main reasons for churn, we can develop targeted strategies to prevent it. For example, pitching two-year contracts to month-to-month customers addresses a specific vulnerability in our current contract terms. Similarly, optimizing payment methods for high-risk customers and implementing paperless billing for low-risk customers are tailored solutions to address identified pain points.

**Conclusion:**

In conclusion, this report highlights the main reasons for customer churn and recommends actionable steps to prevent it. By understanding the impact of various features on customer churn, we can develop targeted strategies to retain our customers and improve overall business performance.
        

This report from business analyst agent went to the Manager who proof read it and made some changes to be released to executive team. Here is the result from Manager agent:

**Main Reasons for Churn and Recommended Actions**

As a manager of data science team, I am pleased to present this enriched report to senior executive teams. Our goal is to identify the main reasons for customer churn and recommend actionable steps to prevent it.

**Main Reasons for Churn:**

1. **Contract Term:** Having a Month-to-month contract increases the likelihood of churn by 16.37%. On the other hand, having a One-year or Two-year contract decreases the probability of churn by 9.98% and 19.89%, respectively.
2. **Payment Method:** Customers using Electronic checks have a higher likelihood of churn (4.30%). In contrast, customers using Bank transfers (automatic) are less likely to churn (-1.65%). The absence of Credit card (automatic) and Mailed check payments also contributes to a lower probability of churn (-2.94% and -4.60%, respectively).
3. **Streaming Services:** Not having Streaming TV or Movies can contribute to a higher likelihood of churn by 1.76% and 0.69%, respectively.
4. **Paperless Billing:** Customers without paperless billing have a lower probability of churn (-3.80%), while those with paperless billing have an increased probability of churn (+1.67%).

**Top 5 Recommended Actions:**

1. **Offer Two-Year Contracts for Month-to-Month Customers:** To reduce the likelihood of churn by 19.89%, offer two-year contracts to customers currently on month-to-month plans.
2. **Promote Electronic Check Payments:** Encourage customers using Mailed checks or Credit cards (automatic) to switch to electronic check payments, which would increase the probability of retention by 4.30%.
3. **Offer Streaming Services:** Provide streaming services to customers without them, as this would decrease the likelihood of churn by 1.76% for TV and 0.69% for movies.
4. **Increase Paperless Billing Adoption:** Encourage customers to adopt paperless billing, which would increase the probability of retention by 3.80%.
5. **Monitor Payment Method Changes:** Keep a close eye on changes in payment methods, as this could indicate a potential churn risk.

**Justification:**

The recommended actions are based on statistical analysis of the provided data. By understanding the relationships between contract terms, payment methods, streaming services, and paperless billing, we can develop targeted strategies to prevent customer churn.

**Conclusion:**

To effectively reduce customer churn, it is essential to understand the underlying factors driving this issue. By analyzing the provided data, we can identify key drivers of churn and recommend actionable steps to prevent it. The top 5 recommended actions outlined in this report aim to address these factors and provide a proactive approach to retaining customers.

**Total Charges:** $X

**Monthly Charges:** $Y

**Payment Method:** [Insert payment method]

**Contract Term:** [Insert contract term]

**Paperless Billing:** [Yes/No]

**Churn Probability:** X%

Recommendations:
1. Offer Two-Year Contracts for Month-to-Month Customers
2. Promote Electronic Check Payments
3. Offer Streaming Services
4. Increase Paperless Billing Adoption
5. Monitor Payment Method Changes
        

Almost the entire points made by these agents were correct after manual inference. This was surprisingly great but at the same time a bit scary too.

Granted this is based on SHAP values and not a causal model. But with domain knowledge there will be many actionable insights from this single report. Imagine this being deployed in an ecosystem with 100s of models, each predicting some value. End users doesn't know what, why or how on these predictions were made. A simple architecture like this, deployed completely locally will be of tremondous help.