Troubleshooting LLM FEN Generation Problems

Claude just published a fascinating technical postmortem that's worth reading if you work with LLMs. Between August and September, three infrastructure bugs were quietly degrading responses. Users started getting random Thai characters mixed into English text. Some requests got routed to servers configured for 1M token contexts when they only needed short ones. Token generation occasionally  just... corrupted.  The interesting part?  Their internal evaluations didn't catch any of it.  Here's what happened:  → 30% of Claude Code users experienced some degraded responses  → At peak, 16% of Sonnet requests were hitting wrong servers→ Some users saw "สวัสดี" randomly appear in English responses  → "Sticky routing" meant if you hit a bad server once, you'd keep hitting it The bugs were caught through user reports, not monitoring. Even with world-class ML infrastructure, the complexity of serving models across multiple hardware platforms (Trainium, GPUs, TPUs) created failure modes their benchmarks couldn't detect. What struck me: this isn't really about preventing LLM errors - they're inevitable in complex distributed systems.  It's about detection and resolution speed.  Some thoughts on LLM reliability: 🔍 Traditional uptime monitoring isn't enough. You need to monitor for "weirdness" - outputs that are technically valid but qualitatively wrong. Think semantic drift, not just HTTP 500s. 👥 User feedback becomes critical infrastructure. Your users often detect issues before your dashboards do. Make reporting easy and act on patterns quickly. ⚡ Consider graceful degradation strategies. Maybe that's fallback models, retry logic with different endpoints, or even hybrid approaches that validate outputs before returning them. The transparency here is refreshing. More companies should share these kinds of deep dives - we all benefit from understanding real-world failure modes. Anyone building LLM applications has stories like this. What's your approach to monitoring model behavior in production?

🪲 Bug Alert! If you’re training an LLM using an on-policy RL algorithm like GRPO, DPO, PPO etc. you should be aware of this bug! 📖 Context Most RL algorithms like GRPO require an LLM to generate completions also called rollouts/trajectories for a given prompt. Most RL libraries leverage inference engines like vLLM or sgLang to compute these rollouts. However when you’re training a model the model whose weights are being updated run on distributed training frameworks like FSDP, DeepSpeed or Nemotron. 🔴 Logit Score Mismatches This discrepancy between training time vs rollout generation time environments has a serious consequence because differences in implementations of different kernels lead to a mismatch in the logit scores of the same set of tokens in a sequence. 🧐 Example So say you have the prompt “Write me a poem about rocks”? Let’s say that the inference engine (vLLM) generates the text “Silent keepers of the earth, …” with some probability/logit scores [P1, P2 P3, P4, P5, ….] for those tokens. Those same tokens for the same prompt will have different prob/logit scores when the model is run outside of vLLM!! 👉🏽 Implications This matters because it means that we would obtain different sets of completions for the same prompt, with a fixed random seed based on which environment the model was running on to obtain the completions. This makes algorithms like GRPO/PPO become slightly off-policy. i.e.. the data that’s used to compute the loss and update the weights is no longer strictly based on the weights of the model being updated! 🔧 Solutions While the ultimate solution is making sure the logit scores of inference engines and train time environments are the same this is a very hard problem to solve. A workaround is importance sampling where you assign a weight to the loss coming from each completion by the ratio between the prob scores obtained in the non-inference env vs inference env. 🎁 Wrap Up Truncated Importance Sampling has recently been implemented by the HF team in TRL to support GRPO. For more read the og blog on this: https://lnkd.in/eg5MpQAe Link to trl docs to enable Importance sampling when ussing vllm: https://lnkd.in/eBH_2Y9t

🤔 🤯 Why does my LLM still give different results at temperature 0? Ever set your LLM temperature to zero, expecting perfectly consistent output, only to get... variations? ⚡⚡You're not alone! This common customer question got me digging deeper. Theory vs. Reality ** In a Perfect World (Temperature 0): The Winner Takes All. The most likely word wins 100% of the time, making other options non-existent. Top-P and Top-K lose their relevance. ** But Real Life is Messy: Implementation details, how LLMs compute probabilities internally, and even tiny computation errors can influence that final “winner.” What Else Impacts Consistency? ** LLM Quirks: Different LLMs may have additional hidden features influencing their output beyond the common temperature, Top-K, and Top-P parameters. ** The Code Matters: Implementation variations within the library you're using can change the outcome slightly. ** Internal Memory: Many LLMs maintain a sort of internal memory, Even a slight tweak in earlier responses could trigger subtle shifts down the line, despite zero temperature. ** Contextual Influence: LLM responses are rarely one-off. During long text generation, subtle changes in how a concept is expressed within the existing text can alter probabilities and the "winning" word choice. ** It's Not Just Words: Behind the curtains, LLMs represent words and concepts as numerical vectors. During computation, the state of these vectors evolves constantly. Minute differences in initialization or calculations could result in the final winning vector for a word receiving a tiny extra push at zero temperature. 💡💡Taming the Wild LLM Here's how to get those outputs under control: ☘️ Seed Phrases: Give your LLM a fixed starting point in a generation task to set a specific direction or enforce a predictable output format..  🧠 Fine-Tuning: Train your model heavily aligned with the desired style and content reduces the model's need to "experiment" for consistency..  👀 Human Oversight: Especially in critical cases, a human editor offers an extra layer of polish.  🤖 Forced Decoding: Instead of sampling, directly instruct the model to output the exact, most probable word at each step. Important Note: True absolute consistency in LLMs can be tricky. Understanding your application's tolerance threshold for such variability is crucial. Image generated via Imagen from Google Cloud #llm #llms #largelanguagemodels #artificialintelligence #thinking #google #googlecloud

Using Pydantic to transform LLM generations into usable dataclasses. I recently integrated an open-source Vision Language Model to extract "structured data" from documents. However, generated outputs were far from being consistent: missing fields, incoherent predictions, JSON format not respected... Using the structured output feature coming with those models, supported by the OpenAI SDK and other LLM providers, helped get better results. Yet, the predictions were in some cases nonsensical. Therefore, I used Pydantic to validate each output. But Pydantic initially serves to validate (or invalidate) the entire schema, leading to an exception error in case of mistakes. To validate each field independently, the solution is to use the BeforeValidator (or field_validator(mode="before") Pydantic feature. As its name suggests, this custom validator occurs before the default Pydantic validator. We can then transform the data based on our needs. If the field is not validated, such as an invoice due date coming before the issued date, or a currency not properly generated (€ instead of EUR format), the validator returns None. And the dataclass output still goes through 👍 . If you work with structured outputs, I can't recommend you enough using this method. It makes the unpredictable nature of LLM consistent for your application. PS: Do you have any other technique?

You're asking the AI to do too much (at once). LLMs struggle with complex, multi-faceted tasks due to competing objectives, broad success criteria, and the cognitive load of handling multiple concerns simultaneously. This leads to higher variance in outputs, reduced reliability, and difficulty in identifying where failures occur in the reasoning chain. Effective AI Engineering Tip #21: Break Complex Tasks into Evaluable Components 👇 The Problem ❌ Many developers approach complex tasks by cramming everything into a single, comprehensive prompt: [Code example - see attached image] Why this approach falls short: - Poor Evaluation Granularity: When the LLM gets sentiment right but categorization wrong, you can't measure individual component performance or identify specific failure points. - Conflicting Objectives: Generating empathetic responses while maintaining factual accuracy creates competing priorities that reduce overall quality. - High Variance: Complex tasks have too many "acceptable" paths to completion, leading to inconsistent outputs and unpredictable behavior. - Difficult Debugging: When something goes wrong, it's nearly impossible to identify which specific component failed or needs improvement. The Solution: Task Decomposition (Prompt Chaining) ✅ A better approach is to break complex tasks into focused, individually evaluable components. This technique, also called "prompt chaining," creates a pipeline where each step has a clear, measurable objective and feeds structured output to the next component. [Code example - see attached image] Why this approach works better: - Individual Evaluation: Each component can be tested and measured separately. Poor categorization doesn't mask good response generation, allowing targeted improvements. - Focused Objectives: Each LLM call has a single, clear purpose, reducing conflicting requirements and improving consistency within each component. - Model Optimization: Use faster, cheaper models (gpt-4o-mini) for structured analysis tasks and reserve powerful models (gpt-4o) for creative response generation. - Easier Debugging: When issues arise, you can pinpoint exactly which component failed and iterate on just that piece without affecting the entire system. The Takeaway ✈️ Don't try to solve complex, multi-objective tasks in a single LLM call. Task decomposition through prompt chaining creates more reliable, evaluable, and maintainable AI systems. Each focused component can be individually optimized, tested, and improved, leading to better overall performance and easier debugging when things go wrong.

Let's talk about key failure points in a Retrieval Augmented Generation (RAG) system as it is one of the most common questions I get. Check out this illustration which I adapted from a popular research paper which summarizes this in a nice visual! Failure points to be aware of when building RAG systems: 1. Missing Content: Means the retrieval strategy didn't work and hence the LLM will reply I don't know or hallucinate 2. Missed Top Ranked: The answer to the question is in your vector DB but did not rank highly enough to be returned to the user 3. Not in Context - Documents with the answer were retrieved from the database but did not make it into the LLM context for generating an answer 4. Not Extracted: Answer is present in the context, but the LLM failed to extract it. Typically, happens when there is noise or contradicting information in the context 5. Wrong Format: Instruction involved generating information in a certain format such as a table or list and the LLM ignored it 6. Incorrect Specificity: Happens when users are too vague about their question or answer is not specific enough or too specific and misses enough background information 7.  Incomplete: Generated answer is correct but misses some of the information even though that information was in the context Some of these issues can be tackled by improving your retrieval strategy as I mentioned in my previous post, using a semantic cache, stronger LLMs and more grounded prompts and plugging in output parsers. However these are still active problems and if you have solved them feel free to share your thoughts! Referenced paper for the illustration and topics is, Seven Failure Points When Engineering a Retrieval Augmented Generation System, do check it out to dive into more details!

You're in a ML Engineer interview at Perplexity, and the interviewer asks: "Your RAG system is hallucinating in production. How do you diagnose what's broken - the retriever or the generator?" Here's how you can answer: Most candidates say "check accuracy" or "run more tests." Wrong approach. RAG systems fail at TWO distinct stages, and you need different metrics for each. Generic accuracy won't tell you WHERE the problem is. The fundamental insight: RAG quality = Retriever Performance × Generator Performance If either component scores zero, your entire system fails. It's multiplication, not addition. You can't compensate for bad retrieval with a better LLM. Retrieval Metrics (Did we get the right context?) 1️⃣ Contextual Relevancy: What % of retrieved chunks actually matter? 2️⃣ Contextual Recall: Did we retrieve ALL the info needed? 3️⃣ Contextual Precision: Are relevant chunks ranked higher than junk? Generation Metrics (Did the LLM use context correctly?) 1️⃣ Faithfulness: Is the output contradicting the retrieved facts? 2️⃣ Answer Relevancy: Is the response actually answering the question? 3️⃣ Custom metrics: Does it follow your specific format/style requirements? btw if you want to receive these bites daily, subscribe my newsletter, and you'll have it in your inbox https://lnkd.in/g8ZJGsWj now back to post - Here's the diagnostic framework every senior ML engineer knows: High faithfulness + Low relevancy = Retrieval problem Low faithfulness + High relevancy = Generation problem Both low = Your entire pipeline is broken Both high = Look for edge cases The metric that catches most production issues: Contextual Recall Your retriever might find "relevant" content but miss critical details. Perfect precision, zero recall = confident wrong answers. This is why RAG systems confidently hallucinate. "Our RAG has 85% accuracy!" Interviewer: "What's your contextual precision? Faithfulness score? Are you measuring end-to-end or component-level?" Vague metrics = You don't understand production RAG systems. The evaluation workflow that separates juniors from seniors: ❌ Junior: Test everything end-to-end, pray it works ✅ Senior: Component-level metrics + automated CI/CD evaluation + production monitoring Know your evaluation targets by use case: Customer support: Faithfulness >0.9 (no wrong info) Research assistant: Contextual recall >0.8 (comprehensive) Code completion: Answer relevancy >0.9 (stay on topic) Legal docs: All metrics >0.95 (zero tolerance) The brutal production reality: Perfect retrieval + weak prompts = hallucinations Perfect LLM + bad chunks = irrelevant answers Good retrieval + good generation + no monitoring = eventual failure You need metrics at ALL stages. Pro tip for the interview: Mention LLM-as-a-judge evaluation. #machinelearning #datascience #genai #ai #gpt #llm #aiagents #inference #rag #evals

While integrating generative AI into financial advisory services at Crediture, I encountered the propensity of LLMs to occasionally 'hallucinate' or generate convincing yet erroneous information. In this article, I share some of the strategies that I had to implement to safeguard against hallucination and protect our users. In summary, they include: ▪ Constrained prompts that scope the capabilities of the LLM to minimize false information generation. ▪ Rigorous testing by conducting invalid input testing with nonsensical prompts to detect over-eagerness in response. ▪ Evaluating confidence scores to filter out low-certainty responses to reduce misinformation risk. Follow Crediture's LinkedIn Page to learn more and keep up with our latest advancements: https://lnkd.in/ggAH79yx

In the context of Large Language Models (LLMs), hallucination refers to the generation of incorrect or misleading information. It occurs when the model produces content that appears plausible but lacks accurate grounding in the given context or data. Mitigating hallucination is crucial for enhancing the reliability and trustworthiness of the outputs generated by LLMs in enterprise applications. This paper succinctly presents 32 techniques aimed at mitigating hallucination in Large Language Models (LLMs). It introduces a taxonomy that categorizes methods, including RAG, Knowledge Retrieval, CoVe, and more. The paper provides practical insights on applying these techniques, emphasizing challenges and inherent limitations. [Link to the paper: arxiv.org/abs/2401.01313]