Managing LLM Variability in Data Analysis

Small variations in prompts can lead to very different LLM responses. Research that measures LLM prompt sensitivity uncovers what matters, and the strategies to get the best outcomes. A new framework for prompt sensitivity, ProSA, shows that response robustness increases with factors including higher model confidence, few-shot examples, and larger model size. Some strategies you should consider given these findings: 💡 Understand Prompt Sensitivity and Test Variability: LLMs can produce different responses with minor rephrasings of the same prompt. Testing multiple prompt versions is essential, as even small wording adjustments can significantly impact the outcome. Organizations may benefit from creating a library of proven prompts, noting which styles perform best for different types of queries. 🧩 Integrate Few-Shot Examples for Consistency: Including few-shot examples (demonstrative samples within prompts) enhances the stability of responses, especially in larger models. For complex or high-priority tasks, adding a few-shot structure can reduce prompt sensitivity. Standardizing few-shot examples in key prompts across the organization helps ensure consistent output. 🧠 Match Prompt Style to Task Complexity: Different tasks benefit from different prompt strategies. Knowledge-based tasks like basic Q&A are generally less sensitive to prompt variations than complex, reasoning-heavy tasks, such as coding or creative requests. For these complex tasks, using structured, example-rich prompts can improve response reliability. 📈 Use Decoding Confidence as a Quality Check: High decoding confidence—the model’s level of certainty in its responses—indicates robustness against prompt variations. Organizations can track confidence scores to flag low-confidence responses and identify prompts that might need adjustment, enhancing the overall quality of outputs. 📜 Standardize Prompt Templates for Reliability: Simple, standardized templates reduce prompt sensitivity across users and tasks. For frequent or critical applications, well-designed, straightforward prompt templates minimize variability in responses. Organizations should consider a “best-practices” prompt set that can be shared across teams to ensure reliable outcomes. 🔄 Regularly Review and Optimize Prompts: As LLMs evolve, so may prompt performance. Routine prompt evaluations help organizations adapt to model changes and maintain high-quality, reliable responses over time. Regularly revisiting and refining key prompts ensures they stay aligned with the latest LLM behavior. Link to paper in comments.

"𝐖𝐡𝐲 𝐢𝐬 𝐦𝐲 𝐋𝐋𝐌 𝐠𝐢𝐯𝐢𝐧𝐠 𝐝𝐢𝐟𝐟𝐞𝐫𝐞𝐧𝐭 𝐚𝐧𝐬𝐰𝐞𝐫𝐬 𝐭𝐨 𝐭𝐡𝐞 𝐬𝐚𝐦𝐞 𝐪𝐮𝐞𝐬𝐭𝐢𝐨𝐧?"  If you have asked this in the last month, here is your Debugging Playbook. Most teams treat inconsistent LLM outputs as a Model Problem.  It is almost never the Model.  It is your System Architecture exposing variability you did not know existed. After debugging 40+ production AI systems, I have developed a 6-Step Framework that isolates the real culprit: Step 1: Confirm the Inconsistency Is Real • Compare responses across identical prompts • Control temperature, top-p, and randomness • Check prompt versions and hidden changes • Goal: Rule out noise before debugging the system Step 2: Break the Output into System Drivers • Decompose your response pipeline into components • Prompt structure, retrieved context (RAG), tool calls, model version, system instructions • Use a "dropped metric" approach to test each driver independently • Goal: Identify where variability can be introduced Step 3: Analyze Variability per Driver • Inspect each driver independently for instability • Does retrieval return different chunks? Are tool outputs non-deterministic? Are prompts dynamically constructed? • Test drivers across same period vs previous period • Goal: Isolate the component causing divergence Step 4: Segment by Execution Conditions • Slice outputs by environment or context • User input variants, model updates/routing, time-based data changes, token limits or truncation • Look for patterns in when inconsistency spikes • Goal: Find conditions where inconsistency spikes Step 5: Compare Stable vs Unstable Runs • Contrast successful outputs with failing ones • Same prompt/different output, same context/different reasoning, same goal/different execution • Surface the exact difference that matters • Goal: Surface the exact difference that matters Step 6: Form and Test Hypotheses • Turn findings into testable explanations • Hypothesis: retrieval drift, prompt ambiguity, tool response variance • Move from suspicion to proof • Goal: Move from suspicion to proof The pattern I see repeatedly: Teams jump straight to "let's try a different model" or "let's add more examples." But inconsistent outputs are rarely a model issue-they are usually a system issue. • Your retrieval is pulling different documents.  • Your tool is returning non-deterministic results.  • Your prompt is being constructed differently based on context length. The 6-step framework forces you to treat LLM systems like the distributed systems they actually are. Which step do most teams skip? Step 1. They assume inconsistency without proving it. Control your variables first. ♻️ Repost this to help your network get started ➕ Follow Anurag(Anu) Karuparti for more PS: If you found this valuable, join my weekly newsletter where I document the real-world journey of AI transformation. ✉️ Free subscription: https://lnkd.in/exc4upeq #GenAI #AIAgents

It shouldn’t surprise people that LLMs are not fully deterministic, they can’t be. Even when you set temperature to zero, fix the seed, and send the exact same prompt, you can still get different outputs in production. There’s a common misconception that nondeterminism in LLMs comes only from sampling strategies. In reality, part of the variability comes from how inference is engineered at scale. In production systems, requests are often batched together to optimize throughput and cost. Depending on traffic patterns, your prompt may be grouped differently at different times. That changes how certain low-level numerical operations are executed on hardware. And because floating-point arithmetic is not perfectly associative, tiny numerical differences can accumulate and lead to different token choices. The model weights haven’t changed, neither has the prompt. But the serving context has. Enterprise teams often evaluate models assuming reproducibility is guaranteed if parameters are fixed. But reliability in LLM systems is not only a modeling problem. It is a systems engineering problem. You can push toward stricter determinism. But doing so may require architectural trade-offs in latency, cost, or scaling flexibility. The point is not that LLMs are unreliable, but that nondeterminism is part of the stack. If you are deploying AI in production, you need to understand where it enters, and design your evaluation, monitoring, and governance around it.

New research from Thinking Machines Lab, the AI Startup of Mira Murati, former OpenAI tech chief. Even with temperature set to 0 (always “pick the top choice”), LLM answers can still vary. It’s not mainly because GPUs are random; it’s because most LLM code changes tiny math details depending on how many requests are processed together (the batch size). When the server is busy, your request gets grouped differently than when it’s quiet, and that tiny change can ripple into a different word later. The fix: write “batch-invariant” kernels, implementations of key ops that give the same result no matter the batch size or slicing strategy, so outputs become truly reproducible. People often say “GPUs run in parallel, so order changes, so results change.” That can happen in some cases, but for the forward pass of today’s LLMs, the commonly used GPU kernels are run-to-run deterministic—they’ll give the same bits if you run the exact same computation the exact same way. So why do you still see different answers? Because you aren’t running “the exact same computation the exact same way.” The way the math is arranged changes based on batch size (how many requests are processed together) and how the sequence is split under the hood. What fixes it? Make core ops batch-invariant so they behave the same no matter server load or how sequences are chunked. In practice that means: - RMSNorm & matmul: use consistent reduction/tile strategies (avoid shape-dependent split-K). - Attention: keep KV layout consistent and use fixed split sizes so reduction order never depends on batch size. Does it work? In their test, Thinking Machines Lab's 1,000 runs at temp 0 produced 80 different completions; with batch-invariant kernels, all 1,000 were identical (with a modest throughput hit). This all matters, for reproducible evals, debuggable regressions, and true on-policy RL (training matches sampling exactly). I'm linking the original paper in the comments.

Achieving 3x-25x Performance Gains for High-Quality, AI-Powered Data Analysis Asking complex data questions in plain English and getting precise answers feels like magic, but it’s technically challenging. One of my jobs is analyzing the health of numerous programs. To make that easier we are building an AI app with Sapient Slingshot that answers natural language queries by generating and executing code on project/program health data. The challenge is that this process needs to be both fast and reliable. We started with gemini-2.5-pro, but 50+ second response times and inconsistent results made it unsuitable for interactive use. Our goal: reduce latency without sacrificing accuracy. The New Bottleneck: Tuning "Think Time" Traditional optimization targets code execution, but in AI apps, the real bottleneck is LLM "think time", i.e. the delay in generating correct code on the fly. Here are some techniques we used to cut think time while maintaining output quality: ① Context-Rich Prompts Accuracy starts with context. We dynamically create prompts for each query: ➜ Pre-Processing Logic: We pre-generate any code that doesn't need "intelligence" so that LLM doesn't have to ➜ Dynamic Data-Awareness: Prompts include full schema, sample data, and value stats to give the model a full view. ➜ Domain Templates: We tailor prompts for specific ontology like "Client satisfaction" or "Cycle Time" or "Quality". This reduces errors and latency, improving codegen quality from the first try. ② Structured Code Generation Even with great context, LLMs can output messy code. We guide query structure explicitly: ➜ Simple queries: Direct the LLM to generate a single line chained pandas expression. ➜ Complex queries : Direct the LLM to generate two lines, one for processing, one for the final result Clear patterns ensure clean, reliable output. ③ Two-Tiered Caching for Speed Once accuracy was reliable, we tackled speed with intelligent caching: ➜ Tier 1: Helper Cache – 3x Faster ⊙ Find a semantically similar past query ⊙ Use a faster model (e.g. gemini-2.5-flash) ⊙ Include the past query and code as a one-shot prompt This cut response times from 50+s to <15s while maintaining accuracy. ➜ Tier 2: Lightning Cache – 25x Faster ⊙ Detect duplicates for exact or near matches ⊙ Reuse validated code ⊙ Execute instantly, skipping the LLM This brought response times to ~2 seconds for repeated queries. ④ Advanced Memory Architecture ➜ Graph Memory (Neo4j via Graphiti): Stores query history, code, and relationships for fast, structured retrieval. ➜ High-Quality Embeddings: We use BAAI/bge-large-en-v1.5 to match queries by true meaning. ➜ Conversational Context: Full session history is stored, so prompts reflect recent interactions, enabling seamless follow-ups. By combining rich context, structured code, caching, and smart memory, we can build AI systems that deliver natural language querying with the speed and reliability that we, as users, expect of it.

Stop loading data into ChatGPT and asking for insights. It is lying to you. An LLM cannot find "truth." It does not know your business context. It does not understand your data. It fabricates plausible narratives and reinforces your confirmation bias. You don't need an insight generator. You need a sparring partner. The LLM's true power is in stress-testing your ideas. This is how you shatter bias. This is how you find the real insight, not just the one you were looking for. Use the "Challenge-Code-Verify" cycle. - The Challenge: State your hypothesis. Command the LLM to act as a skeptical statistician and find 3 ways you are wrong. - The Code: Direct the LLM to produce the exact code (Python/R) or formula (Excel/Sheets) to test its counter-argument. - The Verification: Run the code. Look at the chart. Make the call. This is how you partner with the LLM to sharpen your human abilities -- intuition, creativity, novelty. Asking your LLM for insights is like asking your sparring partner to fight for you. It will get knocked out. Its job isn't to win the match. Its job is to reveal your weaknesses, sharpen your skills, perfect your form, and force you to be better. Spar with your LLM so that when its showtime, you are the one who lands the knockout. Art+Science Analytics Institute | University of Notre Dame | University of Notre Dame - Mendoza College of Business | University of Illinois Urbana-Champaign | University of Chicago | D'Amore-McKim School of Business at Northeastern University | ELVTR | Grow with Google - Data Analytics #Analytics #DataStorytelling

LLMs are great for data processing, but using new techniques doesn't mean you get to abandon old best practices. The precision and accuracy of LLMs still need to be monitored and maintained, just like with any other AI model. Tips for maintaining accuracy and precision with LLMs: • Define within your team EXACTLY what the desired output looks like. Any area of ambiguity should be resolved with a concrete answer. Even if the business "doesn't care," you should define a behavior. Letting the LLM make these decisions for you leads to high variance/low precision models that are difficult to monitor. • Understand that the most gorgeously-written, seemingly clear and concise prompts can still produce trash. LLMs are not people and don't follow directions like people do. You have to test your prompts over and over and over, no matter how good they look. • Make small prompt changes and carefully monitor each change. Changes should be version tracked and vetted by other developers. • A small change in one part of the prompt can cause seemingly-unrelated regressions (again, LLMs are not people). Regression tests are essential for EVERY change. Organize a list of test case inputs, including those that demonstrate previously-fixed bugs and test your prompt against them. • Test cases should include "controls" where the prompt has historically performed well. Any change to the control output should be studied and any incorrect change is a test failure. • Regression tests should have a single documented bug and clearly-defined success/failure metrics. "If the output contains A, then pass. If output contains B, then fail." This makes it easy to quickly mark regression tests as pass/fail (ideally, automating this process). If a different failure/bug is noted, then it should still be fixed, but separately, and pulled out into a separate test. Any other tips for working with LLMs and data processing?

LLM hallucinations present a major roadblock to GenAI adoption (here’s how to manage them) Hallucinations occur when LLMs return a response that is incorrect, inappropriate, or just way off. LLMs are designed to always respond, even when they don’t have the correct answer. When they can’t find the right answer, they’ll just make something up. This is different from past AI and computer systems we’ve dealt with, and it is something new for businesses to accept and manage as they look to deploy LLM-powered services and products. We are early in the risk management process for LLMs, but some tactics are starting to emerge: 1 -- Guardrails: Implementing filters for inputs and outputs to catch inappropriate or sensitive content is a common practice to mitigate risks associated with LLM outputs. 2 -- Context Grounding: Retrieval-Augmented Generation (RAG) is a popular method that involves searching a corpus of relevant data to provide context, thereby reducing the likelihood of hallucinations. (See my RAG explainer video in comments) 3 -- Fine-Tuning: Training LLMs on specific datasets can help align their outputs with desired outcomes, although this process can be resource-intensive. 4 -- Incorporating a Knowledge Graph: Using structured data to inform LLMs can improve their ability to reason about relationships and facts, reducing the chance of hallucinations. That said, none of these measures are foolproof. This is one of the challenges of working with LLMs—reframing our expectations of AI systems to always anticipate some level of hallucination. The appropriate framing here is that we need to manage the risk effectively by implementing tactics like the ones mentioned above. In addition to the above tactics, longer testing cycles and robust monitoring mechanisms for when these LLMs are in production can help spot and address issues as they arise. Just as human intelligence is prone to mistakes, LLMs will hallucinate. However, by putting in place good tactics, we can minimize this risk as much as possible.

I reviewed the literature on the LLM-as-a-judge technique. Here are the key findings. 📉 Model Performance Variability LLMs show inconsistent performance across datasets and tasks. No single model dominates all scenarios. GPT-4 generally leads, with open-source models like Llama-3-70B close behind. 👥 Alignment with Human Judgments LLMs correlate better with non-expert human judgments than expert annotations. Top models approach but don't match human-to-human alignment levels. Improved alignment comes mainly from increased recall, not precision. ⚖️ Evaluation Method Comparison Comparative assessment outperforms absolute scoring in robustness and accuracy. Reference-guided evaluation shows promise but has limitations. Simple methods sometimes unexpectedly outperform complex ones in specific tasks. ⚠️ Vulnerabilities and Biases LLMs struggle with toxicity, safety assessments, and basic perturbations like spelling errors. They show leniency bias and are susceptible to simple adversarial attacks, especially in absolute scoring scenarios. 🛑 Limitations of Fine-tuned Judges Fine-tuned models excel in-domain but lack generalizability and aspect-specific evaluation capabilities. They're prone to superficial biases and don't benefit from prompt engineering techniques (e.g., few-shot prompting). 👩⚖️ LLMs-as-a-jury Using LLMs-as-a-jury (PoLL) outperforms single large judges, reducing bias and cost while improving consistency across tasks. This approach mitigates intra-model favoritism. 💡 Practical Recommendations Use both quantitative metrics and qualitative analysis. Consider perplexity-based detection for adversarial inputs. Multiple judges are better than one. This is a technique used at scale to review models and filter data. It's imperfect, but a poor correlation with human judgment doesn't necessarily mean it's bad. Careful prompt engineering and multiple iterations can give you excellent results in most use cases.

Are your LLM apps still hallucinating? Zep used to as well—a lot. Here’s how we worked to solve Zep's hallucinations. We've spent a lot of cycles diving into why LLMs hallucinate and experimenting with the most effective techniques to prevent it. Some might sound familiar, but it's the combined approach that really moves the needle. First, why do hallucinations happen? A few core reasons: 🔍 LLMs rely on statistical patterns, not true understanding. 🎲 Responses are based on probabilities, not verified facts. 🤔 No innate ability to differentiate truth from plausible fiction. 📚 Training datasets often include biases, outdated info, or errors. Put simply: LLMs predict the next likely word—they don’t actually "understand" or verify what's accurate. When prompted beyond their knowledge, they creatively fill gaps with plausible (but incorrect) info. ⚠️ Funny if you’re casually chatting—problematic if you're building enterprise apps. So, how do you reduce hallucinations effectively? The #1 technique: grounding the LLM in data. - Use Retrieval-Augmented Generation (RAG) to anchor responses in verified data. - Use long-term memory systems like Zep to ensure the model is always grounded in personalization data: user context, preferences, traits etc - Fine-tune models on domain-specific datasets to improve response consistency and style, although fine-tuning alone typically doesn't add substantial new factual knowledge. - Explicit, clear prompting—avoid ambiguity or unnecessary complexity. - Encourage models to self-verify conclusions when accuracy is essential. - Structure complex tasks with chain-of-thought prompting (COT) to improve outputs or force "none"/unknown responses when necessary. - Strategically tweak model parameters (e.g., temperature, top-p) to limit overly creative outputs. - Post-processing verification for mission-critical outputs, for example, matching to known business states. One technique alone rarely solves hallucinations. For maximum ROI, we've found combining RAG with a robust long-term memory solution (like ours at Zep) is the sweet spot. Systems that ground responses in factual, evolving knowledge significantly outperform. Did I miss any good techniques? What are you doing in your apps?