Evaluating LLM Performance Versus Software Reliability

Most people still think of LLMs as “just a model.” But if you’ve ever shipped one in production, you know it’s not that simple. Behind every performant LLM system, there’s a stack of decisions, about pretraining, fine-tuning, inference, evaluation, and application-specific tradeoffs. This diagram captures it well: LLMs aren’t one-dimensional. They’re systems. And each dimension introduces new failure points or optimization levers. Let’s break it down: 🧠 Pre-Training Start with modality. → Text-only models like LLaMA, UL2, PaLM have predictable inductive biases. → Multimodal ones like GPT-4, Gemini, and LaVIN introduce more complex token fusion, grounding challenges, and cross-modal alignment issues. Understanding the data diet matters just as much as parameter count. 🛠 Fine-Tuning This is where most teams underestimate complexity: → PEFT strategies like LoRA and Prefix Tuning help with parameter efficiency, but can behave differently under distribution shift. → Alignment techniques- RLHF, DPO, RAFT, aren’t interchangeable. They encode different human preference priors. → Quantization and pruning decisions will directly impact latency, memory usage, and downstream behavior. ⚡️ Efficiency Inference optimization is still underexplored. Techniques like dynamic prompt caching, paged attention, speculative decoding, and batch streaming make the difference between real-time and unusable. The infra layer is where GenAI products often break. 📏 Evaluation One benchmark doesn’t cut it. You need a full matrix: → NLG (summarization, completion), NLU (classification, reasoning), → alignment tests (honesty, helpfulness, safety), → dataset quality, and → cost breakdowns across training + inference + memory. Evaluation isn’t just a model task, it’s a systems-level concern. 🧾 Inference & Prompting Multi-turn prompts, CoT, ToT, ICL, all behave differently under different sampling strategies and context lengths. Prompting isn’t trivial anymore. It’s an orchestration layer in itself. Whether you’re building for legal, education, robotics, or finance, the “general-purpose” tag doesn’t hold. Every domain has its own retrieval, grounding, and reasoning constraints. ------- Follow me (Aishwarya Srinivasan) for more AI insight and subscribe to my Substack to find more in-depth blogs and weekly updates in AI: https://lnkd.in/dpBNr6Jg

Your unit tests mean nothing for LLM features. assert output == expected That line of code — the foundation of every software test you’ve ever written — is useless the moment your system produces non-deterministic output. And most teams shipping AI features right now have no idea what to replace it with. ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ December 2023. A Chevrolet dealership in California deployed a GPT-4-powered customer service chatbot on their website. Within days, users had prompt-engineered it into agreeing to sell a 2024 Chevy Tahoe — a $58,000 vehicle — for $1. The bot said, and I quote: “that’s a legally binding offer — no takesies backsies.” The screenshots went viral. The model was doing exactly what a poorly evaluated chatbot does: it had no output guardrails, no adversarial testing, and no system checking whether its responses made any sense before they reached customers. This is what happens when you ship an LLM feature with no evaluation pipeline. ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ The most common response from engineers new to LLM work is to reach for BLEU or ROUGE scores. These are the standard NLP metrics — they measure how much the generated text overlaps with a reference answer. They don’t work. Consider these two responses to the same question: Reference: “The server crashed due to a memory leak” Generated: “A memory leak caused the application to go down” These mean the same thing. A human reads both and nods. ROUGE gives the second one a score of 0.22 — nearly zero — because the words don’t overlap. The metric is measuring the wrong thing entirely. ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ What actually works: a three-layer stack. Layer 1 — Deterministic checks. Free, fast, CI-friendly. Does the response refuse when it shouldn’t? Is the JSON valid? Is it hallucinating URLs? These run in milliseconds on every PR. They catch structural failures before anything else. Layer 2 — LLM-as-judge. This sounds circular. You’re using an AI to evaluate an AI. But it works because evaluation is easier than generation. Use pairwise comparison instead of a 1-5 scale — “which response is better, A or B” — and validate that the judge agrees with humans on 50-100 examples before you trust it. Layer 3 — Human review on 2% of traffic. Expensive. Focused on the queries that the automated layers flag as low confidence. ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ The brutal truth: Every prompt change you ship is a regression test you didn’t run. LLM systems fail silently. Your monitoring shows 200 OK and 120ms latency. Meanwhile the model has quietly started refusing queries it handled fine last week. You don’t find out until a user complains. The teams getting this right treat their eval dataset as a first-class artifact alongside their code. Full article — the full three-layer implementation, prompt regression testing in CI Link in comments ↓ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ #SystemDesign #AIEngineering #LLM #MachineLearning

Claude just published a fascinating technical postmortem that's worth reading if you work with LLMs. Between August and September, three infrastructure bugs were quietly degrading responses. Users started getting random Thai characters mixed into English text. Some requests got routed to servers configured for 1M token contexts when they only needed short ones. Token generation occasionally  just... corrupted.  The interesting part?  Their internal evaluations didn't catch any of it.  Here's what happened:  → 30% of Claude Code users experienced some degraded responses  → At peak, 16% of Sonnet requests were hitting wrong servers→ Some users saw "สวัสดี" randomly appear in English responses  → "Sticky routing" meant if you hit a bad server once, you'd keep hitting it The bugs were caught through user reports, not monitoring. Even with world-class ML infrastructure, the complexity of serving models across multiple hardware platforms (Trainium, GPUs, TPUs) created failure modes their benchmarks couldn't detect. What struck me: this isn't really about preventing LLM errors - they're inevitable in complex distributed systems.  It's about detection and resolution speed.  Some thoughts on LLM reliability: 🔍 Traditional uptime monitoring isn't enough. You need to monitor for "weirdness" - outputs that are technically valid but qualitatively wrong. Think semantic drift, not just HTTP 500s. 👥 User feedback becomes critical infrastructure. Your users often detect issues before your dashboards do. Make reporting easy and act on patterns quickly. ⚡ Consider graceful degradation strategies. Maybe that's fallback models, retry logic with different endpoints, or even hybrid approaches that validate outputs before returning them. The transparency here is refreshing. More companies should share these kinds of deep dives - we all benefit from understanding real-world failure modes. Anyone building LLM applications has stories like this. What's your approach to monitoring model behavior in production?

I met an AI PM last week whose agent scored 94% on internal benchmarks. Production users broke it in 48 hours. 𝗧𝗵𝗲 𝗕𝗲𝗻𝗰𝗵𝗺𝗮𝗿𝗸 𝗧𝗿𝗮𝗽 UC Berkeley and Stanford surveyed 306 practitioners shipping agents in production. The data killed a story the industry keeps telling itself. → 74% rely on humans as the primary evaluator → 52% use LLM-as-judge, always with human review → 75% ship without formal benchmarks → 68% cap the agent at 10 steps before a human steps in Autonomous agent hype, meet production reality. 𝗧𝗵𝗲 𝗖𝗼𝘀𝘁 𝗼𝗳 𝗖𝗵𝗮𝗼𝘀 AI PMs optimizing for leaderboard scores are solving the wrong problem. A benchmark doesn't know what your user meant, what compliance allows, or when the agent drifted three turns ago. Scores measure confidence. Users measure correctness. 𝗧𝗵𝗲 𝗔𝗵𝗮 𝗠𝗼𝗺𝗲𝗻𝘁 The best AI PMs stopped asking "what score did it get" and started asking "who caught the failure, how fast, what did we learn." That reframe changes the whole org. 𝗧𝗵𝗲 𝗔𝗿𝗰𝗵𝗶𝘁𝗲𝗰𝘁𝘂𝗿𝗮𝗹 𝗙𝗶𝘅 Five layers every production agent needs: 1 ↳ Domain expert golden sets, built with workflow owners 2 ↳ LLM-as-judge as pre-filter, never the final call 3 ↳ Human verification on every high-stakes path 4 ↳ Real user signals over synthetic scores 5 ↳ Continuous eval, not a pre-launch gate Reliability isn't a benchmark problem. It's a systems problem. The AI PMs getting recruited at 40% premiums aren't posting top MMLU scores. They're posting eval frameworks that survived real users. Benchmarks win demos. Feedback loops win production. Trust isn't automated. It's earned, one verified interaction at a time. PS: Drop EVAL in the comments and I'll send you the golden-set framework I use with AI PMs on my team.

LLM-as-a-Judge is a dangerous way to do evaluation. In most cases, it just doesn't work. While it gives a false impression of having a grasp on your system's performance, it lures you with general metrics such as correctness, faithfulness, or completeness. They hide several complexities: - What does "completeness" mean for your application? In the case of a marketing AI assistant, what characterizes a complete post from an incomplete one? If the score goes higher, does it mean that the post is better? - Often, these metrics are scores between 1-5. But what is the difference between a 3 and a 4? Does a 5 mean the output is perfect? What is perfect, then? - If you "calibrate" the LLM-as-a-judge with scores given by users during a test session, how do you ensure the LLM scoring matches user expectations? If I arbitrarily set all scores to 4, will I perform better than your model? However, if LLM-as-a-judge is limited, it doesn't mean it's impossible to evaluate your AI system. Here are some strategies you can adopt: - Online evaluation is the new king in the GenAI era Log and trace LLM outputs, retrieved chunks, routing… Each step of the process. Link it to user feedback as binary classification: was the final output good or bad? Then take a look at the data. Yourself, no LLM. Take notes and try to find patterns among good and bad traces. At this stage, you can use an LLM to help you find clusters within your notes, not the data. After taking this time, you'll already have some clues about how to improve the system. - Evaluate deterministic steps that come before the final output Was the retrieval system performant? Did the router correctly categorize the initial query? Those steps in the agentic system are deterministic, meaning you can evaluate them precisely: Retriever: Hit Rate@k, Mean Reciprocal Rank, Precision, Recall Router: Precision, Recall, F1-Score Create a small benchmark, synthetically or not, to evaluate offline those steps. It enables you to improve them later on individually (hybrid search instead of vector search, fine-tune a small classifier instead of relying on LLMs…) - Don't use tools that promise to externalize evaluation Your evaluation system is your moat. If you externalize it, not only will you have no control over your AI application, but you also won't be able to improve it. This is your problem, your users, your revenue. Evaluate your system. Not a generic one. All problems are different. Yours is unique as well. ... Those are some unequivocal ideas proposed by the AI community. Yet, I still see AI projects relying on LLM-as-a-judge and generic metrics among companies. Being able to evaluate your system gives you the power to improve it. So take the time to create the perfect evaluation for your use case.

It shouldn’t surprise people that LLMs are not fully deterministic, they can’t be. Even when you set temperature to zero, fix the seed, and send the exact same prompt, you can still get different outputs in production. There’s a common misconception that nondeterminism in LLMs comes only from sampling strategies. In reality, part of the variability comes from how inference is engineered at scale. In production systems, requests are often batched together to optimize throughput and cost. Depending on traffic patterns, your prompt may be grouped differently at different times. That changes how certain low-level numerical operations are executed on hardware. And because floating-point arithmetic is not perfectly associative, tiny numerical differences can accumulate and lead to different token choices. The model weights haven’t changed, neither has the prompt. But the serving context has. Enterprise teams often evaluate models assuming reproducibility is guaranteed if parameters are fixed. But reliability in LLM systems is not only a modeling problem. It is a systems engineering problem. You can push toward stricter determinism. But doing so may require architectural trade-offs in latency, cost, or scaling flexibility. The point is not that LLMs are unreliable, but that nondeterminism is part of the stack. If you are deploying AI in production, you need to understand where it enters, and design your evaluation, monitoring, and governance around it.