Data Analytics for Chatbots

I built a LangGraph-based multi-agent AI system for conversational data analytics capable of ingesting natural language queries and orchestrating context-aware analysis, dynamic visualizations, and flexible data exploration across any tabular dataset. This system demonstrates agent orchestration over stateful message-passing, with each agent encapsulating domain-specific logic and tools. Agents collaborate asynchronously, passing control through a Coordinator node that ensures deterministic execution and robust fallback logic. System Capabilities: - Context-aware query parsing with intelligent routing - Abbreviation expansion and column mapping (“hp” → “horsepower”) - Stateful conversation memory for multi-turn analytics - Dynamic chart generation (bar, violin, scatter, heatmap, etc.) with LLM-powered Python code - Advanced dataframe operations: filtering, grouping, correlation, aggregation - Custom code execution via built-in Python IDE agent - Seamless data search and exploration across arbitrary CSV/Excel files - Persistent session context and query history Agent Topology: - CoordinatorAgent — DAG controller, manages traversal and result aggregation - RouterAgent — classifies intent, routes queries to relevant agents - QueryContextAgent — expands abbreviations, maps query terms, adds context hints - MemoryAgent — maintains chat/session context and formatting - PandasAgent — performs DataFrame/statistical operations - ChartingAgent — LLM-driven code generation for custom visualizations - DataSearchAgent — context-enhanced search and data exploration - PythonIDEAgent — executes safe, custom Python code snippets Tech Stack: - LangGraph: StateGraph-based agent orchestration - LangChain: Agent tools, chain-of-thought, and memory - Streamlit: Web interface for chat-driven analytics - OpenAI GPT-4o-mini: LLM backend for reasoning and code - Pandas/Matplotlib/Seaborn: Data processing & visualization - Python: Modular OOP, TypedDict state containers - Traceability: Internal logging per agent traversal and state Design Rationale: The goal was to build an inspectable, extensible, and production-ready agentic analytics system with real-world applicability. LangGraph’s node-based architecture enables transparent execution, tracing, recovery, and modular agent composition making the design robust, maintainable, and easily extensible to new analytics tasks. The result is a functional architecture for real-time, conversational data analysis separating concerns, maximizing agent interoperability, and minimizing system coupling. Github: https://lnkd.in/gffy62rh

AI chatbots are a dime a dozen these days (or $15 / 1M output tokens if you’re using OpenAI). But building a valuable chatbot takes more than an OpenAI subscription.This story of how the data team at WHOOP used GenAI to democratize access to reliable insights is a masterclass in how to make a useful chatbot. According to Matt Luizzi, his team had “several hundred dashboards and all the typical sprawl you see in BI… Everyone’s creating things, nobody knows what’s being used or what’s correct… Depending on where you go, you may or may not get the right answer.” Matt’s team saw an AI chatbot as the perfect way to create a single source of truth that could be easily—and reliably—queried by his stakeholders. The first order of business? Getting their data quality in order. Here’s how they did it: Step 1. Re-architect their dbt project to improve documentation and accessibility. Step 2. Leverage lineage to deprecate dashboards that weren’t being used. Step 3. Define “golden questions” to audit the chatbot’s outputs. In the end, Matt and his team eliminated 80% of their existing dashboards, and implemented new data quality practices that improved not just the quality of reliability of their chatbot, but the reliability of their broader data platform as well. “Getting in the room and having conversations with the right stakeholders is half the battle,” says Matt. “For us, being able to showcase the fact that we’re able to not only create dashboards and run A/B tests but actually build tooling that’s serving the business — that’s gotten us a lot of value in the organization.” Check out the full story via link in the comments to get all the insights and find out what’s next for the data team at WHOOP.

How to Build an AI Agent for Data Analysis: A Blueprint An "agent" is more than just a chatbot. It’s a system designed to understand a goal, create a plan, and use tools to actively accomplish that goal. You can build your own powerful agent for data analysis, transforming how users interact with their data. This blueprint outlines the core components required to turn simple questions into actionable insights. An agentic system is built on three foundational concepts: an LLM for reasoning, a set of tools for taking action, and a sophisticated memory for learning and context. 1. The LLM: Your Agent's Reasoning Core At the heart of any data analysis agent is its reasoning core: a Large Language Model (LLM) like OpenAI's GPT or Google's Gemini. To build this, create a central orchestrator service (e.g., a Chat Service). This service shouldn't just pass the user's question to the LLM. Instead, it should enrich the prompt with context from the agent's memory. The LLM's role is not merely to respond, but to create a step-by-step plan and generate the precise Python code needed to perform the analysis. 2. Tools: Give Your Agent Hands-on Capabilities An agent is only as good as the tools it can use. For a data analysis agent, the primary tool is the ability to execute code. After the LLM generates an analysis script, your orchestrator service must run it against the relevant dataset. This is the most critical agentic step: it moves the system from simply planning to actively doing. You can equip your agent with other tools, such as services for data loading, chart generation, or even calling external APIs, allowing it to handle a wide variety of analytical tasks. 3. Memory: Enable Context and Learning To elevate your agent from a one-shot tool to an intelligent partner, you need to implement memory. A robust approach is to use a graph database like Neo4j to manage two distinct types: ➜ Short-Term Memory: Implement a mechanism to track the current conversation history for each user session. This allows your agent to understand follow-up questions ("now show me that by region") and maintain context, just like a human analyst would. ➜ Long-Term Memory: This is where your agent can learn. Every time it successfully executes an analysis, store the user's query and the generated code as a "solution." By creating a vector embedding of the query, you can enable semantic search. When a new question comes in, the agent can first search its long-term memory for a similar problem it has already solved, allowing it to deliver accurate results faster and more efficiently over time. By integrating these three components, your application will function as a true AI agent. Your central orchestrator service will drive the powerful loop of Memory -> Reasoning -> Action, creating a system that doesn't just answer questions, but actively solves them.

"Can we build a chatbot that answers questions over our data—like a human analyst?" This is a question I hear often from senior leaders. It sounds simple, but the real answer is more complex than it appears. In my latest article, I unpack why naive prompting approaches often fail when it comes to generating SQL from natural language, and why LLMs alone aren’t enough to deliver consistent, trustworthy results over structured data. Instead, I share how we approached the problem with a 7-layer architecture that combines: Data transformation and semantic modeling, Agentic AI orchestration, Fine-tuned domain models and Strong governance and explainability. I also walk through a real-world deployment involving SQL + NoSQL systems, a custom semantic layer, and vectorized query context to build a reliable analytical chatbot. If your organization is thinking seriously about LLMs for analytics, this might offer a useful perspective.

Most chatbots sound smart. Until you ask a real question. We tested many GenAI tools. Most gave vague or wrong answers. Why? No real understanding of data. Then, we tried something different. We started with the foundation: ✅ Categorize into tuples (Domain, noun, verb) ✅ Clear metadata documentation (table, column, LOVs, etc.) ✅ Consistent naming and types (add aliases and context as required) Then we built prompts on top. Now the bot knows: ➡ What each tuple means ➡ What a “row” really is ➡ How to choose the right field No more guessing. No more fake confidence. Because without context… Even the smartest AI fails. What really made it work? All the behind-the-scenes work: 🔹 Metadata 🔹 Ontology 🔹 Data contracts 🔹 Knowledge graph That’s the part most skip. But that’s where the magic is. Want faster insights? Don’t just build the chatbot. Build the foundation it can trust. #GenBI #DataFoundations #DataDocs #Ontology #KnowledgeGraph #DataProducts #EnterpriseAI #TimeToInsight

Almost every data catalog today comes with a conversational chatbot. The demo usually looks the same: “Show me all tables with customer PII.” The chatbot answers correctly. But this is a very simple question that requires a simple answer. These answers were never hard to get by other means. In addition, these “simple” questions often require users to know quite a bit in advance. To ask “What dashboards depend on the Orders table?” the user must already know the exact table name, the schema, and the technology used to store the Orders table. That’s not intelligent data discovery. What users actually want is help answering complex questions, such as: " If I add or change a column in this transactional table, which downstream pipelines will break? Which reports will be affected, and which exact fields inside those reports?" " If a 3rd party vendor changes a file format (for example, sending a single full_name column instead of separate first_name and last_name fields), which business processes, teams, and decisions will be impacted? How can I reach out to them and inform them about possible breakage? How can I test my changes?" "If I replace this API that consumes data from source A with a new API sourcing data from B, which people, systems, and business processes currently depend on the existing API?" "If the payment processor application changes, which downstream pipelines, services, and operational processes will break or need modification?" For conversational discovery to work in data catalogs, many pieces must come together. Most importantly, metadata must be connected across multiple dimensions, including: 1) Structural and technical metadata: schemas, fields, data types, and storage details 2) Business semantics: definitions, metrics, and rules that explain what the data means 3) End-to-end lineage and impact paths: how data flows and transforms from source to consumption 4) Cross-asset relationships: dependencies across tables, pipelines, data products, APIs, ML models, and dashboards 5) Process context: how data supports business workflows and operational activities 6) People context: ownership, stewardship, and the teams that create, maintain, and rely on the data 7) Usage and behavioral metadata: who uses the data, how often, and for what purpose 8) Operational metadata: freshness, latency, data quality signals If a chatbot can’t navigate these relationships, it can’t support discovery of impact in any meaningful way. Until data catalogs connect metadata across various dimensions, conversational discovery will remain a polished interface built on a shallow metadata foundation.

In the rapidly evolving world of conversational AI, Large Language Model (LLM) based chatbots have become indispensable across industries, powering everything from customer support to virtual assistants. However, evaluating their effectiveness is no simple task, as human language is inherently complex, ambiguous, and context-dependent. In a recent blog post, Microsoft's Data Science team outlined key performance metrics designed to assess chatbot performance comprehensively. Chatbot evaluation can be broadly categorized into two key areas: search performance and LLM-specific metrics. On the search front, one critical factor is retrieval stability, which ensures that slight variations in user input do not drastically change the chatbot's search results. Another vital aspect is search relevance, which can be measured through multiple approaches, such as comparing chatbot responses against a ground truth dataset or conducting A/B tests to evaluate how well the retrieved information aligns with user intent. Beyond search performance, chatbot evaluation must also account for LLM-specific metrics, which focus on how well the model generates responses. These include: - Task Completion: Measures the chatbot's ability to accurately interpret and fulfill user requests. A high-performing chatbot should successfully execute tasks, such as setting reminders or providing step-by-step instructions. - Intelligence: Assesses coherence, contextual awareness, and the depth of responses. A chatbot should go beyond surface-level answers and demonstrate reasoning and adaptability. - Relevance: Evaluate whether the chatbot’s responses are appropriate, clear, and aligned with user expectations in terms of tone, clarity, and courtesy. - Hallucination: Ensures that the chatbot’s responses are factually accurate and grounded in reliable data, minimizing misinformation and misleading statements. Effectively evaluating LLM-based chatbots requires a holistic, multi-dimensional approach that integrates search performance and LLM-generated response quality. By considering these diverse metrics, developers can refine chatbot behavior, enhance user interactions, and build AI-driven conversational systems that are not only intelligent but also reliable and trustworthy. #DataScience #MachineLearning #LLM #Evaluation #Metrics #SnacksWeeklyonDataScience – – –  Check out the "Snacks Weekly on Data Science" podcast and subscribe, where I explain in more detail the concepts discussed in this and future posts:    -- Spotify: https://lnkd.in/gKgaMvbh   -- Apple Podcast: https://lnkd.in/gj6aPBBY    -- Youtube: https://lnkd.in/gcwPeBmR https://lnkd.in/gAC8eXmy

Now that we've decided to start implementing these LLMs, it's time to think about how we'll analyze them. I'd love to hear about how you're evaluating your LLMs, but currently I'm looking at: ✔ Using an LLM to analyze the LLM chat output. If you're working with a chatbot, this might look like standing up a second LLM and using the chat data to ask things like, "What are the top 10 responses where users expressed frustration?" or "Synthesize the customer experience of people using this chatbot." ✔ Using statistical topic modeling to get the most prominent topics users use this chatbot for. I like the pyLDAvis library in Python. There are some great tutorials on Towards Data Science. ✔ The ol' manual inspection and looking to understand if questions are being answered successfully. I would love to hear anything else you're thinking about when analyzing LLM output for the business! I look forward to the day that generated pictures handle words a little better. 😅

🧠 If I were building an AI chatbot with company data, here’s exactly how I’d do it: AI is only as smart as the data you give it. The problem? Most enterprise data is stuck in silos — buried in SQL databases, legacy ERP systems, or custom back-office tools. That’s a huge obstacle if you're trying to build a generative AI chatbot that actually understands your business. Here’s the blueprint I’d use to solve it: 1️⃣ Expose structured data via REST APIs • I’m not manually building these. That’s slow and risky. • I'd use a tool like DreamFactory to instantly auto-generate secure REST APIs from any SQL or NoSQL data source. 2️⃣ Lock down API access •Every endpoint needs role-based access control, API key management, and rate limiting by default. •DreamFactory handles this out-of-the-box. 3️⃣ Connect LLMs through a backend orchestrator • Using LangChain or RAG pipelines, the chatbot can query real-time company data through those APIs. • No more stale knowledge — the bot stays up to date. 4️⃣ Monitor, govern, iterate • Every API call is logged. Every data access is auditable. That’s critical for compliance. 🔒 The real unlock isn’t just AI. It’s secure, scalable access to your data. Most teams focus on the LLM. But the real differentiator is your data plumbing. APIs are the foundation. Security is non-negotiable. Speed is a competitive edge. Curious how this works in production? Reach out! #EnterpriseAI #APIStrategy #ChatbotDevelopment #DreamFactory #DataSecurity #DigitalTransformation