Factors Behind Unreliable LLM Evaluation Processes

Exciting New Research on LLM Evaluation Validity! I just read a fascinating paper titled "LLM-Evaluation Tropes: Perspectives on the Validity of LLM-Evaluations" that addresses a critical issue in our field: as Large Language Models (LLMs) increasingly replace human judges in evaluating information retrieval systems, how can we ensure these evaluations remain valid? The paper, authored by researchers from universities and companies across multiple countries (including University of New Hampshire, RMIT, Canva, University of Waterloo, The University of Edinburgh, Radboud University, and Microsoft), identifies 14 "tropes" or recurring patterns that can undermine LLM-based evaluations. The most concerning trope is "Circularity" - when the same LLM is used both to evaluate systems and within the systems themselves. The authors demonstrate this problem using TREC RAG 2024 data, showing that when systems are reranked using the Umbrela LLM evaluator and then evaluated with the same tool, it creates artificially inflated scores (some systems scored >0.95 on LLM metrics but only 0.68-0.72 on human evaluations). Other key tropes include: - LLM Narcissism: LLMs prefer outputs from their own model family - Loss of Variety of Opinion: LLMs homogenize judgment - Self-Training Collapse: Training LLMs on LLM outputs leads to concept drift - Predictable Secrets: When LLMs can guess evaluation criteria For each trope, the authors propose practical guardrails and quantification methods. They also suggest a "Coopetition" framework - a collaborative competition where researchers submit systems, evaluators, and content modification strategies to build robust test collections. If you work with LLM evaluations, this paper is essential reading. It offers a balanced perspective on when and how to use LLMs as judges while maintaining scientific rigor.

Evaluating LLMs is hard. Evaluating agents is even harder. This is one of the most common challenges I see when teams move from using LLMs in isolation to deploying agents that act over time, use tools, interact with APIs, and coordinate across roles. These systems make a series of decisions, not just a single prediction. As a result, success or failure depends on more than whether the final answer is correct. Despite this, many teams still rely on basic task success metrics or manual reviews. Some build internal evaluation dashboards, but most of these efforts are narrowly scoped and miss the bigger picture. Observability tools exist, but they are not enough on their own. Google’s ADK telemetry provides traces of tool use and reasoning chains. LangSmith gives structured logging for LangChain-based workflows. Frameworks like CrewAI, AutoGen, and OpenAgents expose role-specific actions and memory updates. These are helpful for debugging, but they do not tell you how well the agent performed across dimensions like coordination, learning, or adaptability. Two recent research directions offer much-needed structure. One proposes breaking down agent evaluation into behavioral components like plan quality, adaptability, and inter-agent coordination. Another argues for longitudinal tracking, focusing on how agents evolve over time, whether they drift or stabilize, and whether they generalize or forget. If you are evaluating agents today, here are the most important criteria to measure: • 𝗧𝗮𝘀𝗸 𝘀𝘂𝗰𝗰𝗲𝘀𝘀: Did the agent complete the task, and was the outcome verifiable? • 𝗣𝗹𝗮𝗻 𝗾𝘂𝗮𝗹𝗶𝘁𝘆: Was the initial strategy reasonable and efficient? • 𝗔𝗱𝗮𝗽𝘁𝗮𝘁𝗶𝗼𝗻: Did the agent handle tool failures, retry intelligently, or escalate when needed? • 𝗠𝗲𝗺𝗼𝗿𝘆 𝘂𝘀𝗮𝗴𝗲: Was memory referenced meaningfully, or ignored? • 𝗖𝗼𝗼𝗿𝗱𝗶𝗻𝗮𝘁𝗶𝗼𝗻 (𝗳𝗼𝗿 𝗺𝘂𝗹𝘁𝗶-𝗮𝗴𝗲𝗻𝘁 𝘀𝘆𝘀𝘁𝗲𝗺𝘀): Did agents delegate, share information, and avoid redundancy? • 𝗦𝘁𝗮𝗯𝗶𝗹𝗶𝘁𝘆 𝗼𝘃𝗲𝗿 𝘁𝗶𝗺𝗲: Did behavior remain consistent across runs or drift unpredictably? For adaptive agents or those in production, this becomes even more critical. Evaluation systems should be time-aware, tracking changes in behavior, error rates, and success patterns over time. Static accuracy alone will not explain why an agent performs well one day and fails the next. Structured evaluation is not just about dashboards. It is the foundation for improving agent design. Without clear signals, you cannot diagnose whether failure came from the LLM, the plan, the tool, or the orchestration logic. If your agents are planning, adapting, or coordinating across steps or roles, now is the time to move past simple correctness checks and build a robust, multi-dimensional evaluation framework. It is the only way to scale intelligent behavior with confidence.

I rejected a job offer yesterday. Not because of the salary. Not because of the tech stack. Not even because of the long hours they warned me about. But because, when I asked how they evaluate their AI systems, the hiring manager said: "We just ask it some questions and see if the answers sound right." I stared at them for a moment and realized... They just described the biggest problem in AI today. See, "sounds right" isn't a measurement. It's a hope. Here's what proper LLM evaluation actually looks like: - Accuracy: Can it get factual questions right? (Not 80% of the time. Consistently.) - Hallucination rate: How often does it make things up? (This should be near zero for critical applications.) - Bias metrics: Does it treat all groups fairly? (Measured across demographics, not assumed.) Real Evaluation Frameworks: - BLEU scores for translation quality Perplexity for language modeling Human evaluation with inter-annotator agreement Adversarial testing (red teaming) Domain-specific benchmarks (legal, medical, financial) The Process: > Define success criteria BEFORE deployment > Create diverse test sets (not just happy paths) > Measure consistently across model versions > Track performance over time (models drift) Have humans validate edge cases Why This Matters: Before proper evals: "Our model is amazing!" (based on cherry-picked examples) After proper evals: "Our AI achieves 94.2% accuracy on domain X, with known failure modes Y and Z" The difference? One builds trust. The other destroys it when reality hits. The kicker: Most companies are still in the "sounds right" phase. They're deploying models evaluated by vibes, not metrics. Just like you wouldn't join a team that deploys code without tests, you shouldn't join one that deploys AI without proper evaluation. What's your experience with LLM evaluation? Are we measuring what actually matters? #AI #LLM #machinelearning #datascience #job #hiring #chatgpt #gpt4 #gpt5

I spent the past year primarily focused on the emerging field of AI Evalution and successfully applying it in production. My notes on why it is hard... 1. AI quality has no stable definition. Quality shifts with context, intent, and domain nuance. It forces teams to rethink what “good” means every time the use case changes. 2. AI evaluation is failure-first. LLMs don’t fail in generic ways; they fail in random patterns hyper-specific to each use case. You can’t predict these failure modes in advance. You have to uncover them through real examples and trace analysis. 3. AI systems shift constantly. Data, retrieval, users, and upstream dependencies all drift over time. The result is a moving target where validators need continual rechecking and adjustment. 4. Ground truth varies, and much of it is subjective. Some tasks have crisp correctness, others rely on expert interpretation. This pushes evaluation into a mix of methods, since no single metric works across all outputs. 5. Human judgment is essential but doesn’t scale. Experts catch subtle errors instantly, but can’t review thousands of examples. This creates a gap between the quality we can recognize and the volume we need to measure. 6. Automated evaluators introduce their own errors. LLM-as-judge drifts, hallucinates, and inherits model biases. Left unchecked, it can create new blind spots or faulty flagging within the evaluation process. 7. Evaluation must reflect the product, not the model. Every feature has its own definition of “good,” shaped by workflow, risk tolerance, domain rules, and user expectations. Quality means something different at each layer of the product, so evaluation standards have to adapt to those nuances. A lot of this work plays to strengths we already have in data science, research, ml, and subject matter experts. But AI evaluation is not the traditional plug and play product analytics playbook. It introduces a new level of rigor around an exploratory mindset. #ai #evaluation #datascience #ml #productdevelopment #datadriven #datadrivenproddev

When I launched my first GenAI feature, I had to completely relearn how to define "Is it good enough to launch"? I was comfortable with Traditional ML metrics. If you asked me about Precision, Recall, or F1 Scores, I could have a great discussion with Data Scientists about whether or not we are ready to launch. But when my engineering lead asked me for the Go/No-Go criteria for our new LLM feature, those metrics didn't help me much. He asked a simple question: "How do we know this is good enough to ship?" To be honest? At first I didn't know what to answer... I realized I could speak "Traditional ML Metrics", but I didn't know how to speak "LLM Quality". My initial strategy was what most teams do: Vibe Check it and launch because there's pressure from an executive. We ran a few files through the model, read the outputs, and nodded. "Yeah, looks good. Ship it!" 🚀 That works for a prototype. (Don't do that in production...) We tend to test for the 'Happy Paths' that we know our LLM can do, and tend to ignore thoroughly testing all the other things that might break your feature. Through that launch (and a lot of research since), I learned that you need a mix of three specific evaluation layers to actually trust your launch: ⚡ 1. Code-Based Evals (Sanity Checks) Start here - Use standard code (Python/Regex) to catch the "dumb" errors instantly like... - Did the model return valid JSON? - Is the answer under 50 words? It’s instant and free. It doesn't tell you if the answer is smart, but it tells you if it's broken. 🧠 2. Human Evals (Your Ground Truth) Most teams try to skip this layer because it is slow, expensive, and manual, and they want to jump straight to automation. It's a trap! You need real humans to grade the outputs to define what "Good" looks like. This creates your Golden Dataset—the source of truth that you measure everything else against. If you don't do this, your automated metrics are just measuring noise. The toughest part about Human Evals is convincing your stakeholders (esp. leadership) that you need to spend ALL THAT TIME doing Human Evaluations - they are critical, don't skip them. 🤖 3. LLM-as-a-Judge (Fast and Scalable) Ideally, you'll use a stronger model to grade your own model’s output. Something like... - Input: "Compare the Model's answer to the Golden Answer." - Criteria: "Rate accuracy on a scale of 1-5." This lets you run thousands of tests in minutes. It allows you to scale your "human" logic without the human bottleneck and keep evaluating as you make changes. The biggest mistake you can make is thinking you can automate trust. You can't. You have to put in the hard, manual work (Human Evals) too. --- 👋 How are you currently measuring quality for your AI features? Are you using "vibes" or do you have a formal eval pipeline? --- 💎 Need help with Evals? George Zoto, Marily Nika, Ph.D and I put together a hands-on AI Evals course for you - Check my comment below for a code with a discount!

What actually happens when LLMs evaluate LLM-generated research? 🐍 Scientific quality quietly collapses. New research analyzing 125,000+ paper-review pairs from ICLR, NeurIPS, and [ICML] Int'l Conference on Machine Learning just dropped on arXiv, and the findings are a wake-up call for scientific integrity. When LLMs review research papers, the core problem isn’t hallucination. It’s Rating Compression. LLM reviewers are trained to be helpful and polite. That makes them very bad at giving strong rejections and strong endorsements. Everything gets squeezed into a beige middle - grammatically perfect, low-variance, low-conviction reviews. This creates three dangerous illusions: → It looks like LLM reviewers prefer LLM-written papers. In reality, weaker papers tend to use more AI writing, and LLM reviewers are simply too “nice” to flag mediocrity. → The signal that separates breakthrough research from plausible-sounding work disappears. → Worst of all, LLM-assisted meta-reviews are significantly more likely to flip a decision to “Accept” given the same underlying scores than a human meta-reviewer would. If we use LLMs to write papers and to grade papers, we don’t just lose the human touch - we lose the ability to distinguish insight from polish. Some takeaways that I found useful... • Authors: If you use an LLM to pre-review your paper, ignore the score. Focus only on critiques of logic, novelty, and assumptions. • Reviewers: Watch for beige reviews, polished language with no strong stance on novelty or impact. • Chairs: High confidence + low variance is classic bot behavior. Evaluation systems need variance-aware checks. This isn’t about banning LLMs from peer review. It’s about understanding their systematic biases and designing processes that compensate for them. As an engineer, I love automation, especially when there are 20k+ submissions. But judgment? That still has to stay human. IMO LLMs can assist, but should never be the final arbiter. Curious where others draw the line 💭 #AIEthics #PeerReview #ICML #ICLR #NeurIPS

This Nature Medicine paper is not an indictment of users. It’s an indictment of how we evaluate and deploy LLMs. The study shows something subtle but important: when large language models are used as public-facing medical assistants, performance collapses—not because people are “bad users,” but because the systems are not designed to function reliably in real human interactions. In controlled testing, the models themselves perform well. But once embedded in an interactive setting, their outputs become: 1. inconsistent across semantically similar inputs 2. poorly calibrated for decision-making 3. difficult for non-experts to interpret or act on safely That gap is not a user failure. It’s a design and evaluation failure. Standard benchmarks (medical exams) and even simulated users systematically overestimate real-world safety. They measure stored knowledge, not whether a system can reliably guide action under uncertainty. And medical care is always about managing uncertainty. Humans do what humans always do: provide partial information reason under ambiguity rely on cues like consistency and clarity If an AI system degrades under those conditions, the responsibility lies with the system—not the person using it. For high-stakes domains like healthcare, “human-in-the-loop” is not a safety guarantee. Interaction itself is the risk surface. Until models are designed, tested, and regulated around real user behavior, benchmark performance will remain a misleading proxy for safety. https://lnkd.in/epT2YaEM #AI #Medicine #patients #humans

We've all shipped an LLM feature that "felt right" in dev, only to watch it break in production. Why? Because human "eyeballing" isn't a scalable evaluation strategy. The real challenge in building robust AI isn't just getting an LLM to generate an output. It’s ensuring the output is 𝐫𝐢𝐠𝐡𝐭, 𝐬𝐚𝐟𝐞, 𝐟𝐨𝐫𝐦𝐚𝐭𝐭𝐞𝐝, 𝐚𝐧𝐝 𝐮𝐬𝐞𝐟𝐮𝐥, consistently, across thousands of diverse user inputs. This is where 𝐄𝐯𝐚𝐥𝐮𝐚𝐭𝐢𝐨𝐧 𝐌𝐞𝐭𝐫𝐢𝐜𝐬 become non-negotiable. Think of them as the sophisticated unit tests and integration tests for your LLM's brain. You need to move beyond "does it work?" to "how well does it work, and why?" This is precisely what Comet's 𝐎𝐩𝐢𝐤 is designed for. It provides the framework to rigorously grade your LLM's performance, turning subjective feelings into objective data. Here's how we approach it, as shown in the cheat sheet below: 1./ Heuristic Metrics => the 'Linters' & 'Unit Tests' - These are your non-negotiable, deterministic sanity checks. - They are low-cost, fast, and catch objective failures. - Your pipeline should fail here first. ▫️Is it valid? → IsJson, RegexMatch ▫️Is it faithful? → Contains, Equals ▫️Is it close? → Levenshtein 2./ LLM-as-a-Judge => the 'Peer Review' - This is for everything that "looks right" but might be subtly wrong. - These metrics evaluate quality and nuance where statistical rules fail. - They answer the hard, subjective questions. ▫️Is it true? → Hallucination ▫️Is it relevant? → AnswerRelevance ▫️Is it helpful? → Usefulness 3./ G-Eval => the dynamic 'Judge-Builder' - G-Eval is a task-agnostic LLM-as-a-Judge. - You define custom evaluation criteria in plain English (e.g., "Is the tone professional but not robotic?"). - It then uses Chain-of-Thought reasoning internally to analyze the output and produce a human-aligned score for those criteria. - This allows you to test specific business logic without writing new code. 4./ Custom Metrics - For everything else. - This is where you write your own Python code to create a metric. - It’s for when you need to check an output against a live internal API, a proprietary database, or any other logic that only your system knows. Take a look at the cheat sheet for a quick breakdown. Which metric are you implementing first for your current LLM project? ♻️ Don't forget to repost.