Xiao Han

Evaluating Large Language Models (LLMs) is no longer just about accuracy scores. As LLMs move into production and power real-world applications, evaluation must balance scale, nuance, and trust. Automatic Evaluation is the first layer. It relies on predefined metrics and usually a ground-truth reference, making it fast and cost-effective for large datasets and objective tasks. Metrics like BLEU (n-gram overlap for translation), ROUGE (content recall for summarization), exact match/accuracy (factual Q&A), and semantic similarity (embedding-based meaning comparison) help quantify model performance where correctness is clearly defined. LLM-as-a-Judge (LLMaaJ) goes a step further. Here, a separate and often stronger LLM evaluates model outputs against qualitative criteria. This approach captures aspects traditional metrics miss—such as helpfulness, coherence, relevance, tone, and completeness—and works even without reference answers. It is especially powerful for RAG systems, where the judge can assess both retrieval quality (context recall/precision) and generation quality (faithfulness and answer relevance). Custom metrics also allow evaluation aligned to domain or brand-specific requirements. Human-in-the-Loop (HITL) review remains critical. For highly subjective judgments, complex edge cases, and final production validation, human reviewers provide the gold standard. This is essential for assessing creativity, empathy, cultural fit, and domain-critical accuracy, especially in regulated areas like healthcare or legal, where expert curation helps uncover failure modes that automated systems cannot. In practice, strong LLM evaluation is layered—automatic metrics for scale, LLM-based judgment for depth, and human review for confidence and accountability. #LLMEvaluation #GenAI #AIEngineering #AgenticAI #RAG #MLOps #ResponsibleAI #AIQuality

6

Fine-tuning AI shouldn't require thousands of labeled examples. Reinforcement Fine-Tuning (RFT) for Amazon Nova changes that. Just give it prompts + a reward function. The model learns through feedback — not imitation. ✅ No massive datasets ✅ Works for code, customer service, financial reasoning ✅ Runs on Amazon Bedrock, SageMaker, or Nova Forge Smarter customization. Less grunt work. I wrote a deep-dive on this — check out my blog 👇 https://lnkd.in/ggNNezuC Authors:me, along with Chakravarthy Nagarajan Bharathan Balaji Vignesh Radhakrishnan #AWS #GenerativeAI #AmazonNova #MachineLearning #LLM

44

I just published Vol. 123 of "Top Information Retrieval Papers of the Week" on Substack. My Substack newsletter features the 7-10 most notable research papers on information retrieval (including recommender systems, search & ranking, etc.) from each week, with a brief summary, and links to the paper/codebase. This week’s newsletter highlights the following research work: 📚 Scalable Multimodal Retrieval Through Flexible Late Interaction and Test-Time Budget Control, from Meta 📚 How Interaction Noise Shapes Embedding Scalability in Collaborative Filtering Models, from He et al. 📚 Embedding Domain Expertise in LLMs through Reinforcement Learning from Augmented Generation, from Nie et al. 📚 EmbeddingGemma: Powerful and Lightweight Text Representations, from Google DeepMind 📚 Bringing Context Engineering and Reasoning to Industrial Cascade Ranking Systems, from Shopee 📚 The Role of Vocabularies in Learning Sparse Representations for Ranking, from Naver 📚 Improving Dual Encoders for Multi-Level Document Retrieval, from Google DeepMind 📚 A Production-Ready Generative Framework for Unified Search and Recommendation, from Alibaba 📚 A Unified PyTorch Framework for Large-Scale Sparse-Dense Recommendation Training, from Alibaba 📚 Treating Reranking as Noise Reduction in Multi-Stage Recommender Systems, from Kuaishou #InformationRetrieval #ResearchPapers #CuratedContent #Newsletter  #substack

27

2 Comments

Today I spoke with two frinds from Google AI and learned that the KPI for engineers at Google and many large tech companies has fundamentally changed. A key metric now is how many tokens you are able to use. The more tokens you use, the greater your potential contribution—one person can now accomplish the work that once required ten or even a hundred people. #HerPowerAI #Google #WorkModel #KPI

1

Do we get some a free DGX Spark depending on the number of companies we have worked in our career that showed up today during Jensen Huang's NVIDIA GTC keynote? 🤣