How to Assess Fine-Tuned Language Models

Researchers from Salesforce AI have just unveiled SFR-RAG, a groundbreaking 9B parameter language model that's pushing the boundaries of contextual understanding and retrieval-augmented generation (RAG)! SFR-RAG-9B outperforms larger models like Command-R+ (104B) on multiple benchmarks, achieving SOTA results in 3 out of 7 tasks in the newly introduced ContextualBench evaluation suite. The model excels at faithful comprehension of provided contexts, minimizing hallucination, and handling unanswerable or counterfactual scenarios. To create a contextually faithful language model like SFR-RAG, here are the key steps: 1. Design a novel chat template   - Introduce "Thought" and "Observation" roles in addition to standard System, User, and Assistant roles.   - Use "Thought" for internal reasoning and tool-use syntax.   - Use "Observation" for external information and function call results. 2. Prepare training data   - Synthesize diverse instruction-following data mimicking real-world retrieval QA applications.   - Include scenarios for extracting information from long contexts, handling unanswerable queries, recognizing conflicting information, and dealing with distracting or out-of-distribution content. 3. Fine-tune the model   - Use supervised fine-tuning and preference learning techniques.   - Train on the prepared instruction-following dataset.   - Focus on context-grounded generation and hallucination minimization. 4. Implement function-calling capabilities   - Train the model to use external tools and perform multi-hop reasoning.   - Incorporate strategies similar to Self-RAG, ReAct, and other agentic approaches. 5. Evaluate the model   - Use ContextualBench, a compilation of 7 popular RAG and contextual benchmarks.   - Ensure a consistent evaluation setup across all tasks.   - Measure performance using multiple metrics (Exact Match, Easy Match, F1 score). 6. Test for resilience   - Evaluate the model's performance on the FaithEval suite.   - Test its ability to handle unknown, conflicting, and counterfactual information. 7. Assess general capabilities   - Evaluate on standard LM benchmarks (e.g., MMLU, GSM8K).   - Test function-calling abilities using the Berkeley function-calling benchmark. 8. Iterate and refine   - Analyze results and identify areas for improvement.   - Adjust training data, fine-tuning processes, or model architecture as needed.

LLM-as-a-Judge (LaaJ) and reward models (RMs) are similar concepts, but understanding their nuanced differences is important for applying them correctly in practice… LLM-as-a-Judge is a reference-free evaluation metric that assesses model outputs by simply prompting a powerful language model to perform the evaluation for us. In the standard setup, we ask the model to either: - Provide a direct assessment score (e.g., binary or Likert score) of a model’s output. - Compare the relative quality of multiple outputs (i.e., pairwise scoring). There are many choices for the LLM judge we use. For example, we can use an off-the-shelf foundation model, fine-tune our own model, or form a "jury" of several LLM judges. Reward models are specialized LLMs—usually derived from the LLM we are currently training—that are trained to predict a human preference score given a prompt and a candidate completion as input. A higher score from the RM indicates higher human preference. Similarities between LaaJ and RMs: Both LaaJ and RMs can provide direct assessment and pairwise (preference) scores. Therefore, both techniques can be used for evaluation. Given these similarities, recent research has explored combining RMs and LaaJ into a single model with both capabilities. Differences between LaaJ and RMs: Despite their surface similarities, these two techniques have many fundamental differences: - RMs are fine-tuned using a preference learning or ranking objective, whereas fine-tuned LaaJ models usually learn via standard language modeling objectives. - LaaJ models are often based on off-the-shelf or foundation LLMs, whereas RMs are always fine-tuned. - LaaJ is based on a standard LLM architecture, while RMs typically add an additional classification head to predict a preference score. - RMs only score single model outputs (though we can derive a preference score by plugging multiple RM scores into a preference model like Bradley-Terry), whereas LaaJ can support arbitrary scoring setups (i.e., is more flexible). Where should we use each technique? Given these differences, recent research has provided insights into where LaaJ and RMs are most effective. LaaJ should be used for evaluation purposes (both direct assessment and pairwise). This is an incredibly powerful evaluation technique that is used almost universally. When we compare the evaluation accuracy of LaaJ (assuming correct setup and tuning) to RMs, LaaJ models tend to have superior scoring accuracy; for example, in RewardBench2, LaaJ models achieve the highest accuracy on pairwise preference scoring. Despite LaaJ’s strengths, RMs are still more useful for RL-based training with LLMs (e.g., PPO-based RLHF). Interestingly, even though LaaJ models provide more accurate preference scores, they cannot be directly used as RMs for RL training. It is important that the RM is derived from the policy currently being trained, meaning we must train a custom RM based on our current policy for RLHF to work properly.

An important lesson from working with hundreds of customers on LLM deployments: there's a **big difference** in how to evaluate and fine-tune language models based on whether your task has **one right answer** or **many**. Let me explain why this matters. Tasks with one correct answer (let's call them "deterministic") include things like classification, structured extraction, and Copilot flows that produce a single action. These are cases where you can quickly check if an output is objectively correct. In contrast, "freeform" tasks have infinitely many valid outputs - think summaries, email drafts, and chatbots. Here, correctness is more subjective, with no single "right" answer. Looking at 1,000 recent datasets on OpenPipe: ~63% were freeform ~37% deterministic. Interestingly though, among the highest-volume tasks, 60% were deterministic - likely because machine-consumed outputs tend to run at higher volume. This distinction drives three key differences in implementation: 1️⃣ Deterministic tasks usually need temperature=0 for consistent, correct outputs. Freeform tasks benefit from higher temperatures (0.7-1.0) to enable creativity and variety. 2️⃣ evaluation approaches differ. Deterministic tasks can use "golden datasets" with known-correct outputs. Freeform tasks often need vibe checks, LLM-as-judge approaches, or direct user feedback. 3️⃣ fine-tuning strategies diverge. For deterministic tasks, Reinforcement Fine-Tuning (RFT) shows promise when correctness is verifiable. For freeform tasks, preference-based methods like DPO or RLHF work better for guiding style and tone. Some practical tips for deterministic tasks: - Consider smaller, specialized models for classification/extraction - Use logprobs to measure classification confidence - You can often reduce costs significantly by going small without losing accuracy For freeform tasks: - Use DPO to train on pairs of good/bad outputs - Consider RLHF to optimize for real user feedback or business metrics - Focus on measuring and improving subjective quality The key is matching your approach to your use case. Don't automatically reach for the largest, most expensive model - sometimes a smaller, more focused solution works better! Lots more details and examples in my post here: https://lnkd.in/gFWdA7kr

Hedge your AI investments the smart way, for instance, before a big fine-tuning effort. Before allocating compute to fine-tune a model, here’s how to reduce risk and maximize upside. If you’re building GenAI systems (RAG, agents, multimodal), you’ve probably felt the urge to fine-tune. But the real question is: should you? And if yes, how do you hedge the cost, time, and model drift risks? Here’s the framework I would recommend for teams evaluating domain-specific fine-tuning: 1. Start orchestration-first Before any finetune, orchestrate multiple base models with retrieval or prompt engineering. This gets you faster results and a testbed to compare performance across off-the-shelf LLMs. 2. Set metrics gates before you train Don’t fall in love with the idea of fine-tuning. Fall in love with the business lift. Here's a quick & dirty playbook for this - Define Gate A (pilot): measurable KPI improvement over a baseline. - Define Gate B (scale): sustained lift + stable cost/ unit + safety. - Define Gate C (prod): passes A/B tests + beats orchestration fallback. If your model doesn’t pass these? Revert. Don’t ship sunk costs to prod. 3. Use fine-tuning as a test, not a commitment Treat fine-tuning as an experiment with a rollback plan. - Choose LoRA or QLoRA for fast iteration - Evaluate on real downstream tasks - Always benchmark against the orchestrated baseline tested with synthetic data generated when getting stuck. No safety or cost wins? Keep orchestration. 4. Build your fallback first Your fine-tuned model will degrade over time. Latency might spike. Hallucinations might sneak in. So build a fallback orchestration layer from day one. This gives you production stability, no matter how your fine-tuned model behaves. Takeaway: Fine-tuning can be powerful – but only when your baseline orchestration shows promise, your metrics are beating thresholds, and you’ve hedged against drift, cost, and time-to-value.

Most people still think of LLMs as “just a model.” But if you’ve ever shipped one in production, you know it’s not that simple. Behind every performant LLM system, there’s a stack of decisions, about pretraining, fine-tuning, inference, evaluation, and application-specific tradeoffs. This diagram captures it well: LLMs aren’t one-dimensional. They’re systems. And each dimension introduces new failure points or optimization levers. Let’s break it down: 🧠 Pre-Training Start with modality. → Text-only models like LLaMA, UL2, PaLM have predictable inductive biases. → Multimodal ones like GPT-4, Gemini, and LaVIN introduce more complex token fusion, grounding challenges, and cross-modal alignment issues. Understanding the data diet matters just as much as parameter count. 🛠 Fine-Tuning This is where most teams underestimate complexity: → PEFT strategies like LoRA and Prefix Tuning help with parameter efficiency, but can behave differently under distribution shift. → Alignment techniques- RLHF, DPO, RAFT, aren’t interchangeable. They encode different human preference priors. → Quantization and pruning decisions will directly impact latency, memory usage, and downstream behavior. ⚡️ Efficiency Inference optimization is still underexplored. Techniques like dynamic prompt caching, paged attention, speculative decoding, and batch streaming make the difference between real-time and unusable. The infra layer is where GenAI products often break. 📏 Evaluation One benchmark doesn’t cut it. You need a full matrix: → NLG (summarization, completion), NLU (classification, reasoning), → alignment tests (honesty, helpfulness, safety), → dataset quality, and → cost breakdowns across training + inference + memory. Evaluation isn’t just a model task, it’s a systems-level concern. 🧾 Inference & Prompting Multi-turn prompts, CoT, ToT, ICL, all behave differently under different sampling strategies and context lengths. Prompting isn’t trivial anymore. It’s an orchestration layer in itself. Whether you’re building for legal, education, robotics, or finance, the “general-purpose” tag doesn’t hold. Every domain has its own retrieval, grounding, and reasoning constraints. ------- Follow me (Aishwarya Srinivasan) for more AI insight and subscribe to my Substack to find more in-depth blogs and weekly updates in AI: https://lnkd.in/dpBNr6Jg

Training a Large Language Model (LLM) involves more than just scaling up data and compute. It requires a disciplined approach across multiple layers of the ML lifecycle to ensure performance, efficiency, safety, and adaptability. This visual framework outlines eight critical pillars necessary for successful LLM training, each with a defined workflow to guide implementation: 𝟭. 𝗛𝗶𝗴𝗵-𝗤𝘂𝗮𝗹𝗶𝘁𝘆 𝗗𝗮𝘁𝗮 𝗖𝘂𝗿𝗮𝘁𝗶𝗼𝗻: Use diverse, clean, and domain-relevant datasets. Deduplicate, normalize, filter low-quality samples, and tokenize effectively before formatting for training. 𝟮. 𝗦𝗰𝗮𝗹𝗮𝗯𝗹𝗲 𝗗𝗮𝘁𝗮 𝗣𝗿𝗲𝗽𝗿𝗼𝗰𝗲𝘀𝘀𝗶𝗻𝗴: Design efficient preprocessing pipelines—tokenization consistency, padding, caching, and batch streaming to GPU must be optimized for scale. 𝟯. 𝗠𝗼𝗱𝗲𝗹 𝗔𝗿𝗰𝗵𝗶𝘁𝗲𝗰𝘁𝘂𝗿𝗲 𝗗𝗲𝘀𝗶𝗴𝗻: Select architectures based on task requirements. Configure embeddings, attention heads, and regularization, and then conduct mock tests to validate the architectural choices. 𝟰. 𝗧𝗿𝗮𝗶𝗻𝗶𝗻𝗴 𝗦𝘁𝗮𝗯𝗶𝗹𝗶𝘁𝘆 and 𝗢𝗽𝘁𝗶𝗺𝗶𝘇𝗮𝘁𝗶𝗼𝗻: Ensure convergence using techniques such as FP16 precision, gradient clipping, batch size tuning, and adaptive learning rate scheduling. Loss monitoring and checkpointing are crucial for long-running processes. 𝟱. 𝗖𝗼𝗺𝗽𝘂𝘁𝗲 & 𝗠𝗲𝗺𝗼𝗿𝘆 𝗢𝗽𝘁𝗶𝗺𝗶𝘇𝗮𝘁𝗶𝗼𝗻: Leverage distributed training, efficient attention mechanisms, and pipeline parallelism. Profile usage, compress checkpoints, and enable auto-resume for robustness. 𝟲. 𝗘𝘃𝗮𝗹𝘂𝗮𝘁𝗶𝗼𝗻 & 𝗩𝗮𝗹𝗶𝗱𝗮𝘁𝗶𝗼𝗻: Regularly evaluate using defined metrics and baseline comparisons. Test with few-shot prompts, review model outputs, and track performance metrics to prevent drift and overfitting. 𝟳. 𝗘𝘁𝗵𝗶𝗰𝗮𝗹 𝗮𝗻𝗱 𝗦𝗮𝗳𝗲𝘁𝘆 𝗖𝗵𝗲𝗰𝗸𝘀: Mitigate model risks by applying adversarial testing, output filtering, decoding constraints, and incorporating user feedback. Audit results to ensure responsible outputs. 🔸 𝟴. 𝗙𝗶𝗻𝗲-𝗧𝘂𝗻𝗶𝗻𝗴 & 𝗗𝗼𝗺𝗮𝗶𝗻 𝗔𝗱𝗮𝗽𝘁𝗮𝘁𝗶𝗼𝗻: Adapt models for specific domains using techniques like LoRA/PEFT and controlled learning rates. Monitor overfitting, evaluate continuously, and deploy with confidence. These principles form a unified blueprint for building robust, efficient, and production-ready LLMs—whether training from scratch or adapting pre-trained models.