Online Performance Evaluation Methods

Your AI model scored 95%. Your users still hate it. Most teams building AI products are stuck at offline evals, testing models against fixed datasets before real users ever touch them. The scores go up. Leadership feels good. But Mario Rodriguez, CPO of GitHub, calls out what actually happens: teams build incentive systems to pass the test, not improve the product (Episode 223). “When a measure becomes a target, it stops being useful” (Goodhart's Law). The discipline nobody talks about is moving from offline to online evaluations and measuring what users actually do in production. At GitHub Copilot, they track two metrics: AR (acceptance rate, did the developer accept the suggestion?) and ARC (accepted and retained characters, how much of that code did they actually keep?). A developer might accept a 20-line suggestion, then immediately rewrite 18 of those lines. Offline evals would score that as success. Production data tells the real story. Mario's advice? Expect offline and online performance to diverge. Don't panic when it happens. Build the online measurement infrastructure early, before you convince yourself the offline score means you're done. Are you measuring model performance or actual user value?

LLM-as-a-Judge is a dangerous way to do evaluation. In most cases, it just doesn't work. While it gives a false impression of having a grasp on your system's performance, it lures you with general metrics such as correctness, faithfulness, or completeness. They hide several complexities: - What does "completeness" mean for your application? In the case of a marketing AI assistant, what characterizes a complete post from an incomplete one? If the score goes higher, does it mean that the post is better? - Often, these metrics are scores between 1-5. But what is the difference between a 3 and a 4? Does a 5 mean the output is perfect? What is perfect, then? - If you "calibrate" the LLM-as-a-judge with scores given by users during a test session, how do you ensure the LLM scoring matches user expectations? If I arbitrarily set all scores to 4, will I perform better than your model? However, if LLM-as-a-judge is limited, it doesn't mean it's impossible to evaluate your AI system. Here are some strategies you can adopt: - Online evaluation is the new king in the GenAI era Log and trace LLM outputs, retrieved chunks, routing… Each step of the process. Link it to user feedback as binary classification: was the final output good or bad? Then take a look at the data. Yourself, no LLM. Take notes and try to find patterns among good and bad traces. At this stage, you can use an LLM to help you find clusters within your notes, not the data. After taking this time, you'll already have some clues about how to improve the system. - Evaluate deterministic steps that come before the final output Was the retrieval system performant? Did the router correctly categorize the initial query? Those steps in the agentic system are deterministic, meaning you can evaluate them precisely: Retriever: Hit Rate@k, Mean Reciprocal Rank, Precision, Recall Router: Precision, Recall, F1-Score Create a small benchmark, synthetically or not, to evaluate offline those steps. It enables you to improve them later on individually (hybrid search instead of vector search, fine-tune a small classifier instead of relying on LLMs…) - Don't use tools that promise to externalize evaluation Your evaluation system is your moat. If you externalize it, not only will you have no control over your AI application, but you also won't be able to improve it. This is your problem, your users, your revenue. Evaluate your system. Not a generic one. All problems are different. Yours is unique as well. ... Those are some unequivocal ideas proposed by the AI community. Yet, I still see AI projects relying on LLM-as-a-judge and generic metrics among companies. Being able to evaluate your system gives you the power to improve it. So take the time to create the perfect evaluation for your use case.

⚖️ 𝗟𝗟𝗠 𝗮𝘀 𝗔 𝗝𝘂𝗱𝗴𝗲 𝗳𝗼𝗿 𝗔𝘂𝘁𝗼𝗺𝗮𝘁𝗶𝗰 𝗘𝘃𝗮𝗹𝘂𝗮𝘁𝗶𝗼𝗻 𝗘𝘃𝗮𝗹𝘂𝗮𝘁𝗶𝗻𝗴 𝘁𝗵𝗲 𝗽𝗲𝗿𝗳𝗼𝗿𝗺𝗮𝗻𝗰𝗲 of machine learning systems is one of the most time-consuming yet critical steps in development. But what if we could automate that? 🤖 𝗟𝗟𝗠-𝗮𝘀-𝗮-𝗝𝘂𝗱𝗴𝗲 — a powerful, and often underestimated, use case of large language models. Instead of only generating content, LLMs can evaluate outputs: 𝗮𝘀𝘀𝗶𝗴𝗻𝗶𝗻𝗴 𝘀𝗰𝗼𝗿𝗲𝘀, 𝗰𝗼𝗺𝗽𝗮𝗿𝗶𝗻𝗴 𝗮𝗹𝘁𝗲𝗿𝗻𝗮𝘁𝗶𝘃𝗲𝘀, 𝗼𝗿 𝗲𝘃𝗲𝗻 𝗴𝗶𝘃𝗶𝗻𝗴 𝗮 𝘀𝗶𝗺𝗽𝗹𝗲 ✅ pass / ❌ fail verdict. 💡 𝗪𝗵𝘆 𝘁𝗵𝗶𝘀 𝗺𝗮𝘁𝘁𝗲𝗿𝘀: ⚡ 𝗙𝗮𝘀𝘁𝗲𝗿 𝗶𝘁𝗲𝗿𝗮𝘁𝗶𝗼𝗻 → Automating evaluations reduces manual review time. 🎯 𝗛𝗶𝗴𝗵𝗲𝗿 𝗿𝗲𝗹𝗶𝗮𝗯𝗶𝗹𝗶𝘁𝘆 → Standardized evaluation criteria make results more consistent. 💰 𝗖𝗼𝘀𝘁 𝘀𝗮𝘃𝗶𝗻𝗴𝘀 → While LLM calls aren’t free, using them strategically can dramatically reduce human evaluation cycles. 🔑 𝗧𝗵𝗿𝗲𝗲 𝗖𝗼𝗿𝗲 𝗘𝘃𝗮𝗹𝘂𝗮𝘁𝗶𝗼𝗻 𝗠𝗲𝘁𝗵𝗼𝗱𝘀 1️⃣ 𝗖𝗼𝗺𝗽𝗮𝗿𝗲 𝘁𝘄𝗼 𝗼𝘂𝘁𝗽𝘂𝘁𝘀 Useful when testing prompt variations, different models, or RAG embeddings. The 𝗟𝗟𝗠 𝗷𝘂𝗱𝗴𝗲 decides whether outputs are equal, or which one is better. 2️⃣ 𝗦𝗰𝗼𝗿𝗲 𝗼𝘂𝘁𝗽𝘂𝘁𝘀 (1–10 or simplified scale) Ideal for experiments with multiple prompt versions or models. Anchoring with example scores improves accuracy. 3️⃣ 𝗣𝗮𝘀𝘀/𝗙𝗮𝗶𝗹 𝗰𝗵𝗲𝗰𝗸𝘀 Especially powerful in RAG systems — did the answer correctly reflect the retrieved context? Clear definitions and few-shot examples improve reliability. 📝 𝗞𝗲𝘆 𝗖𝗼𝗻𝘀𝗶𝗱𝗲𝗿𝗮𝘁𝗶𝗼𝗻𝘀 👥 𝗛𝘂𝗺𝗮𝗻 𝗖𝗼𝗺𝗽𝗮𝗿𝗶𝘀𝗼𝗻 → Always benchmark against human evaluators to ensure alignment. Blind tests are best. 💸 𝗖𝗼𝘀𝘁 𝗔𝘄𝗮𝗿𝗲𝗻𝗲𝘀𝘀 → Frequent evaluations can add up. Use cheaper models for bulk checks or reduce test sizes. 🔧 𝗔𝗱𝗮𝗽𝘁𝗮𝗯𝗶𝗹𝗶𝘁𝘆 → No single method fits all. The right evaluation strategy depends on your system (QA, classification, extraction, etc.). 🚀 𝗪𝗵𝘆 𝗜𝘁’𝘀 𝗣𝗼𝘄𝗲𝗿𝗳𝘂𝗹 Think about deploying a new prompt in production. Instead of manually checking hundreds of responses, you can let the LLM judge decide whether the new version performs as well — or better — than the old one. If results hold, deploy confidently. ✅ 𝗙𝗶𝗻𝗮𝗹 𝗧𝗮𝗸𝗲𝗮𝘄𝗮𝘆 𝗟𝗟𝗠-𝗮𝘀-𝗮-𝗝𝘂𝗱𝗴𝗲 isn’t just a clever trick; it’s a scalable evaluation framework. By offloading repetitive validation to LLMs, teams can move faster, reduce bottlenecks, and still maintain quality. 𝗜𝘁’𝘀 𝗻𝗼𝘁 𝗽𝗲𝗿𝗳𝗲𝗰𝘁 — careful alignment with human evaluators is essential — but it’s a tool every AI practitioner should have in their arsenal. 🔹 Have you experimented with 𝗟𝗟𝗠𝘀 𝗮𝘀 𝗲𝘃𝗮𝗹𝘂𝗮𝘁𝗼𝗿𝘀 in your workflows? What challenges or benefits have you seen? #AI #LLM #MachineLearning #GenerativeAI #ArtificialIntelligence #AICommunity #RAG #AgenticAI #Automation

PART 2: HR Workflows getting automated using Agents. WORKFLOW 2: Performance Review Automation using Agents. The Problem Traditional performance management is inefficient, time-consuming, and often biased. Employees and managers rely on manual reviews, subjective assessments, and incomplete data from scattered sources. This leads to inconsistent feedback, lack of actionable insights, and limited growth opportunities for employees. Solution: An AI-powered Performance Management System automates data collection, feedback analysis, and performance evaluation. By aggregating inputs from multiple sources: self-assessments, manager feedback, chat logs, meeting summaries, and psychometric insights, the system provides a holistic, unbiased, and data-driven performance report. 1) The system first gathers inputs from self-assessments, manager feedback, HR 1:1 meeting notes, and structured performance review frameworks, ensuring a holistic view of an employee’s contributions. 2) AI-driven agents further enhance this process by analyzing Slack messages, Zoom interactions, 1:1 feedback, and psychometric evaluations, providing a deeper and more comprehensive understanding of employee performance trends. 3) Once data is collected, the Performance Report Analysis Agent processes it using company-specific performance guidelines. The Employee Performance Analyst Agent continuously monitors this information, delivering real-time feedback, identifying skill gaps, and suggesting personalized goal-setting strategies that align with business objectives. 4) Finally, automated performance reporting and coaching streamline HR’s role in talent development. The Review & Report Generator Agent compiles structured performance reports that outline employee strengths, areas for improvement, and career development recommendations. Complementing this, an AI Coach provides employees with personalized coaching insights, helping them better understand their strengths and weaknesses while offering guidance for professional growth. 5) This AI-driven workflow not only enhances the efficiency and accuracy of performance evaluations but also empowers employees with actionable  insights for career development, fostering a more engaged and high-performing workforce. Tech Stack: LLMs: openAI, GPT4-0 Data Sources: Google Forms/Spreadsheets, Slack, Zoom, HR platforms Vector Database: Qdrant Agent Framework: Lyzr AI Agent API Hosting: AWS Agents: Performance Report Analysis Agent, Slack Messages Analysis Agent, Zoom Meetings Analysis Agent, 1:1 Feedback Analysis Agent, Psychometric Analysis Agent, Employee Performance Analyst Agent, Review & Report Generator Agent. #HRAgents

We've analyzed over 30,000 performance reviews at Confirm, and what we found is consistently alarming. 50% of companies' top performers AND toxic employees remain completely hidden from leadership. 64% of employees view these reviews as a partial or complete waste of time. These aren't just statistics to me. They're personal. BACKGROUND: I've spent 20 years watching talented people get overlooked while office politicians get promoted. I've seen hundreds of HOURS wasted on calibration meetings filled with bias and politics. I've watched companies lose top performers because they couldn't identify who was truly mission-critical. And I've felt the frustration of realizing traditional performance reviews reflect who you know, not your impact. THE PROBLEM: Traditional reviews are fundamentally broken in today's networked world of work: 1) They rely on single-manager assessment (a single point of failure). 2) They reward visibility over actual impact. 3) They're biased toward people who "play the game" well. 4) They fail to identify your true high performers (who often leave quietly). 5) They're administratively exhausting for everyone involved. HOW ONA CHANGES EVERYTHING: Unlike traditional methods, Organizational Network Analysis (ONA) leverages your entire organization instead of just cherry-picked peers. By asking research-validated questions like "Who do you go to for help?" and "Who's making outstanding impact?", we don’t just measure the “What” of work, but also the “How” that managers often don’t see directly. It's like putting on infrared goggles in a dark room. Suddenly you can see what was always there but invisible: your quiet innovators, your connectors, your true top performers. THE IMPACT: One customer, Thoropass, aimed to avoid losing key talent during the Great Resignation. Using ONA, they identified their true mission-critical employees (many who weren't on leadership's radar!)—and retained 100% of them over 12 months. Another, Canada Goose, discovered 2.5X more top performers than their traditional process had surfaced. THE REALITY: Performance reviews weren't designed for today's hybrid, collaborative, cross-functional world. The most impactful employees often work behind the scenes, connecting teams and solving problems others don't even see. ONA finally gives these people the recognition they deserve while giving leaders the confidence to make fair, data-driven talent decisions. We've built an entire system to make this easy. Because data-driven recognition of true impact isn't some far-off ideal—it's rapidly becoming the new normal. The era of highly subjective, opinion-driven talent decisions needs to end.

LLM Model Evaluation: Offline vs. Online Evaluating large language models is one of the most important steps in building trustworthy AI. A strong evaluation process helps us launch better models and improve them over time for real-world users. 🔎 Why evaluate LLMs? Model evaluation checks if the outputs are accurate, relevant, and safe It uncovers both strengths and blind spots before real users are impacted A careful evaluation process builds user trust and saves time fixing problems later 🧪 Offline Evaluation (Pre-Launch): Uses test sets, curated prompts, and synthetic data Runs in controlled conditions where “correct” answers are known Good for benchmarks, regression testing, and catching bugs before release Results are quick and repeatable Cannot fully predict how real users will interact or what edge cases they find 🌐 Online Evaluation (In Production) Tracks model performance using real user queries and live traffic Captures surprises, new patterns, and issues that slip through static tests Enables A/B testing, live monitoring, and detecting performance drift Helps teams react quickly and improve models based on user needs Needs careful setup to prevent risky outputs from reaching users ⚖️ How to make the most of both Start with a strong offline test suite that evolves with your use case Implement robust dashboards and alerting for live behavior Compare both sets of results often to reveal hidden gaps or new trends Use insights from each evaluation type to tune and strengthen your models 💡 How are you approaching LLM evaluation? What’s worked or not worked for your team? Please share your experience or questions in the comments.

To run good evals, you need to write good scorers. Here's how to write better ones, based on what we've been seeing: Use binary scoring for LLM judges: Ask yes/no questions like "Does this response answer the user's question? 1 for yes, 0 for no." Binary scoring is easier to debug when it breaks. Write separate scorers for each quality dimension: Don't try to measure quality in one scorer. Build individual scorers for accuracy, tone, format compliance, and safety, then combine with weighted averages. Start with code-based checks then layer in LLM judges: JSON validation, length limits, and regex patterns can be handled with code first. Then use LLM judges only for subjective criteria that code can't capture. Include examples in your judge prompts: Show the model what good vs bad looks like. Enable chain-of-thought reasoning: Make your LLM judge explain its decision before scoring. You'll be more likely to catch logical errors and understand when the judge is confused. Test scorers on edge cases: Your first scorer will work on happy path examples. Test it on weird inputs, empty responses, and adversarial cases to build a representative test dataset that covers different user personas (you can use Loop to generate these). Run the same scorers offline and online: Use identical evaluation logic during development and in production to create a feedback loop so production edge cases improve your offline testing. Review low-scoring outputs all the time: When outputs score poorly, figure out why. Could be things like missing criteria, bad prompts, or other edge cases you didn't consider. Read our docs for more: https://lnkd.in/g_dCs7Nh