Improving LLM Training for Niche Applications

One of the biggest barriers to deploying LLM-based agents in real workflows is their poor performance on long-horizon reasoning. Agents often generate coherent short responses but struggle when a task requires planning, tool use, or multi-step decision-making. The issue is not just accuracy at the end, but the inability to reason through the middle. Without knowing which intermediate steps helped or hurt, agents cannot learn to improve. This makes long-horizon reasoning one of the hardest and most unsolved problems for LLM generalization. It is relatively easy for a model to retrieve a document, answer a factual question, or summarize a short email. It is much harder to solve a billing dispute that requires searching, interpreting policy rules, applying edge cases, and adjusting the recommendation based on prior steps. Today’s agents can generate answers, but they often fail to reflect, backtrack, or reconsider earlier assumptions. A new paper from Google DeepMind and Stanford addresses this gap with a method called SWiRL: Step-Wise Reinforcement Learning. Rather than training a model to get the final answer right, SWiRL trains the model to improve each step in a reasoning chain. It does this by generating synthetic multi-step problem-solving traces, scoring every individual step using a reward model (Gemini 1.5 Pro), and fine-tuning the base model to favor higher-quality intermediate steps. This approach fundamentally changes the way we train reasoning agents. Instead of optimizing for final outcomes, the model is updated based on how good each reasoning step was in context. For example, if the model generates a search query or a math step that is useful, even if the final answer is wrong, that step is rewarded and reinforced. Over time, the agent learns not just to answer, but to reason more reliably. This is a major departure from standard RLHF, which only gives feedback at the end. SWiRL improves performance by 9.2 percent on HotPotQA, 16.9 percent on GSM8K when trained on HotPotQA, and 11 to 15 percent on other multi-hop and math datasets like MuSiQue, BeerQA, and CofCA. It generalizes across domains, works without golden labels, and outperforms both supervised fine-tuning and single-step RL methods. The implications are substantial: we can now train models to reason better by scoring and optimizing their intermediate steps. Better reward models, iterative reflection, tool-assisted reasoning, and trajectory-level training will lead to more robust performance in multi-step tasks. This is not about mere performance improvement. It shows how we can begin to train agents not to mimic outputs, but to improve the quality of their thought process. That’s essential if we want to build agents that work through problems, adapt to new tasks, and operate autonomously in open-ended environments.

Retrieval-Augmented Reasoning Modeling Introduces RARE, a new paradigm for training domain-specific LLMs that focuses on reasoning, not memorization. Key ideas: • Inspired by Bloom’s Taxonomy – RARE shifts LLM training from memorizing knowledge (“Remember”) to applying and evaluating it (“Analyze”, “Create”). It separates domain knowledge (retrieved externally) from domain thinking (learned during training), enabling better performance under tight parameter budgets. • Open-book prepared training – RARE injects retrieved knowledge into training prompts, letting models learn reasoning patterns instead of rote facts. This open-book, reasoning-first setup beats both standard SFT and RAG approaches, especially in medicine. • Massive accuracy gains with small models – On five medical QA benchmarks, RARE-trained Llama-3.1-8B and Qwen-2.5-7B outperformed GPT-4 + RAG, with up to +20% accuracy boosts (e.g., PubMedQA: 78.63% vs. GPT-4’s 75.2%, CoVERT: 74.14% vs. GPT-4’s 65.67%). • Training via distillation + adaptive retries – RARE distills answers (and reasoning paths) from a strong teacher (e.g., QwQ-32B), refining outputs until a correct answer is found. This creates a high-quality dataset that teaches contextualized, case-based thinking. • New role for retrieval – Unlike standard RAG (used only at inference), RARE uses retrieval during training to shape reasoning. It models knowledge integration (p(k|x, R(x))) and reasoning (p(r|x, R(x), k)) as separate steps, replacing memorization with application. Overall, this work reframes LLM training for domain-specific intelligence: externalize facts, internalize reasoning. It unlocks strong performance from small models without overfitting or hallucination.

Fine-tuning for making expert, domain-specific models? Not so fast! I often get asked whether companies should fine-tune LLMs to internalize the knowledge required for their particular use case or domain. The answer I give is probably not…. There is research suggesting that large language models struggle to acquire new factual knowledge through fine-tuning. Novel knowledge is learned more slowly than knowledge consistent with what the model already knows. This same research also showed that when knowledge is eventually learned from novel examples, there is a linear increase in the model's tendency to hallucinate. Ouch! So what can you do? What should you do? RAG is one approach, but that comes with complexity and its own challenges: RAG pipelines are more complex, with larger storage costs, higher memory and compute requirements (due to longer contexts demanded by the additional context) and higher latency, due to the need to query an external index. In the long term, storing knowledge natively in the model's parameters may also provide generalization advantages, as the model can relate different pieces of knowledge in its parameters. This is particularly apparent for complex or indirect queries, where simple retrieval augmentation may fall short. A very exciting recent paper from Meta introduced a new approach called Active Reading. This approach leverages synthetic data to have LLMs generate a range of diverse training data based on a closed body of knowledge. By having the LLMs read and restructure the data in many and varied ways and training on that enlarged, restructured corpus, you can significantly improve the model's retention of the contained facts. Active Reading applies the same principles observed in human studying, allowing the model itself to propose multiple study strategies — e.g., paraphrasing, knowledge linking, active recall, etc. — and instantiates these different strategies on a document-by-document basis. This process results in a highly diverse and contextually grounded signal which can then be trained on. The authors demonstrate huge gains vs. vanilla fine-tuning: +313% and +160% (relative improvement over vanilla fine-tuning) on SimpleQA and FinanceBench respectively. They also trained a SOTA 8B model for factual QA, demonstrating the utility of the technique at pre-training scale (1T tokens). It should be noted that the Active Reading paper focuses on knowledge acquisition; that traditional fine tuning can still be useful for instilling style, format, reasoning patterns, or other behaviors. Learning Facts at Scale with Active Reading https://lnkd.in/e7FCAq-3 Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations? https://lnkd.in/e_REAVZB

After about a year and a half working with LLMs I've seen a few tips on how to turn a commercial LLM into your in-house expert: my six-step playbook is below: 1️⃣ Pick the lightest customization that does the job: • Retrieval-Augmented Generation keeps the base model frozen and pipes in your own documents at run time. • Fine-tuning bakes stable expertise directly into the weights. • Hybrid approaches freeze what rarely changes and retrieve what does. 2️⃣ Obsess over data quality: Clean, permission-cleared text matters more than GPU hours. Redact PII, keep training chunks under two thousand tokens, and label a handful of gold-standard examples for every task. 3️⃣ Choose a training method that matches your budget: Full fine-tune for “mission-critical or bust,” Low-Rank Adaptation (LoRA) when you have one GPU and a deadline, instruction tuning for conversational agents, reinforcement learning if safety and tone need tight control. 4️⃣ Stand up an evaluation pipeline before launch: Automated test suites (DeepEval, RAGAs, MLflow Evaluate) score every new checkpoint for accuracy, relevance, bias, and hallucination. Treat prompts like code: unit-test them nightly. 5️⃣ Build guardrails in, not on: Add content filters, prompt-injection shields, and telemetry hooks that log inputs, outputs, and confidence scores. Compliance teams sleep better when monitoring is automatic. 6️⃣ Iterate in production: Canary releases send five percent of traffic to the new model and compare KPIs. Active-learning loops capture low-confidence answers and route them back into the next training batch. Schedule quarterly refreshes so improvement is routine, not heroic. Key takeaway: start with data and evaluation, layer on the lightest customization path that meets accuracy, and measure everything. Do that, and your “off-the-shelf” LLM will start speaking your organization’s language in record time. What’s your go-to tactic for customizing large language models? Drop it below so we can all learn faster. Thoughts?

Training a Large Language Model (LLM) involves more than just scaling up data and compute. It requires a disciplined approach across multiple layers of the ML lifecycle to ensure performance, efficiency, safety, and adaptability. This visual framework outlines eight critical pillars necessary for successful LLM training, each with a defined workflow to guide implementation: 𝟭. 𝗛𝗶𝗴𝗵-𝗤𝘂𝗮𝗹𝗶𝘁𝘆 𝗗𝗮𝘁𝗮 𝗖𝘂𝗿𝗮𝘁𝗶𝗼𝗻: Use diverse, clean, and domain-relevant datasets. Deduplicate, normalize, filter low-quality samples, and tokenize effectively before formatting for training. 𝟮. 𝗦𝗰𝗮𝗹𝗮𝗯𝗹𝗲 𝗗𝗮𝘁𝗮 𝗣𝗿𝗲𝗽𝗿𝗼𝗰𝗲𝘀𝘀𝗶𝗻𝗴: Design efficient preprocessing pipelines—tokenization consistency, padding, caching, and batch streaming to GPU must be optimized for scale. 𝟯. 𝗠𝗼𝗱𝗲𝗹 𝗔𝗿𝗰𝗵𝗶𝘁𝗲𝗰𝘁𝘂𝗿𝗲 𝗗𝗲𝘀𝗶𝗴𝗻: Select architectures based on task requirements. Configure embeddings, attention heads, and regularization, and then conduct mock tests to validate the architectural choices. 𝟰. 𝗧𝗿𝗮𝗶𝗻𝗶𝗻𝗴 𝗦𝘁𝗮𝗯𝗶𝗹𝗶𝘁𝘆 and 𝗢𝗽𝘁𝗶𝗺𝗶𝘇𝗮𝘁𝗶𝗼𝗻: Ensure convergence using techniques such as FP16 precision, gradient clipping, batch size tuning, and adaptive learning rate scheduling. Loss monitoring and checkpointing are crucial for long-running processes. 𝟱. 𝗖𝗼𝗺𝗽𝘂𝘁𝗲 & 𝗠𝗲𝗺𝗼𝗿𝘆 𝗢𝗽𝘁𝗶𝗺𝗶𝘇𝗮𝘁𝗶𝗼𝗻: Leverage distributed training, efficient attention mechanisms, and pipeline parallelism. Profile usage, compress checkpoints, and enable auto-resume for robustness. 𝟲. 𝗘𝘃𝗮𝗹𝘂𝗮𝘁𝗶𝗼𝗻 & 𝗩𝗮𝗹𝗶𝗱𝗮𝘁𝗶𝗼𝗻: Regularly evaluate using defined metrics and baseline comparisons. Test with few-shot prompts, review model outputs, and track performance metrics to prevent drift and overfitting. 𝟳. 𝗘𝘁𝗵𝗶𝗰𝗮𝗹 𝗮𝗻𝗱 𝗦𝗮𝗳𝗲𝘁𝘆 𝗖𝗵𝗲𝗰𝗸𝘀: Mitigate model risks by applying adversarial testing, output filtering, decoding constraints, and incorporating user feedback. Audit results to ensure responsible outputs. 🔸 𝟴. 𝗙𝗶𝗻𝗲-𝗧𝘂𝗻𝗶𝗻𝗴 & 𝗗𝗼𝗺𝗮𝗶𝗻 𝗔𝗱𝗮𝗽𝘁𝗮𝘁𝗶𝗼𝗻: Adapt models for specific domains using techniques like LoRA/PEFT and controlled learning rates. Monitor overfitting, evaluate continuously, and deploy with confidence. These principles form a unified blueprint for building robust, efficient, and production-ready LLMs—whether training from scratch or adapting pre-trained models.

If you’re building LLM applications today, reasoning is where the real leverage lies. And yet, I see a lot of engineers still treating LLM outputs as a single-shot black box. LLMs can reason, but only if you give them the right scaffolding and the right post-training. Here’s a mental model I’ve been using to think about LLM reasoning methods (see chart below): ✅ Inference-time reasoning methods: These are techniques that can be applied at inference time, without needing to retrain your model: → Tree of Thoughts (ToT), search through reasoning paths → Chain of Thought (CoT) prompting, prompt models to generate intermediate reasoning steps → Reasoning + Acting, use tools or function calls during reasoning → Self-feedback, prompt the model to critique and refine its own output → Episodic Memory Agents, maintain a memory buffer to improve multi-step reasoning → Self-consistency, sample multiple reasoning paths and select the most consistent answer ✅ Training-time enhancements: Where things get really powerful is when you post-train your model to improve reasoning, using human annotation or policy optimization: → Use Preference pairs and Reward Models to tune for better reasoning (RFT, Proximal PO, KL Regularization) → Apply RLHF, PPO + KL, Rejection Sampling + SFT, Advantage Estimation, and other advanced techniques to guide the model’s policy → Leverage multiple paths, offline trajectories, and expert demonstrations to expose the model to rich reasoning signals during training Here are my 2 cents 🫰 If you want production-grade LLM reasoning, you’ll need both, → Smart inference-time scaffolds to boost reasoning without slowing latency too much → Carefully tuned post-training loops to align the model’s policy with high-quality reasoning patterns → We’re also seeing increasing use of Direct Preference Optimization (DPO) and reference-free grading to further improve reasoning quality and stability. I’m seeing more and more teams combine both strategies, and the gap between "vanilla prompting" and "optimized reasoning loops" is only getting wider. 〰️〰️〰️ Follow me (Aishwarya Srinivasan) for more AI insight and subscribe to my Substack to find more in-depth blogs and weekly updates in AI: https://lnkd.in/dpBNr6Jg

Day 10/30 of LLMs/SLMs - Pre-training, Fine-Tuning and PEFT Pretraining is where it all begins. The model learns language itself i.e. syntax, semantics, reasoning, by predicting the next word across vast amounts of data - books, code, Wikipedia, and the web. This phase is heavy: hundreds of billions of tokens, thousands of GPUs, weeks of training.  Models like GPT, LLaMA, and Mistral all start this way. Fine-tuning is where the model gets focused. You take that pretrained generalist and train it on your data - medical records, product descriptions, customer chats - so it speaks your domain’s language. Now the problem? Full fine-tuning can be computationally expensive. Especially when models have billions of parameters. Also updating all of the parameters for every task is wasteful. Enter PEFT. Instead of retraining the entire model, PEFT techniques tweak only a small subset of parameters, like <1%, while keeping the rest frozen. Three popular methods are - 👉 LoRA (Low-Rank Adaptation): Adds tiny trainable matrices to attention layers. It’s like giving your model small adjustment knobs rather than rewiring the whole circuit. Hugely popular and performant compared to the rest for domain adaptation (e.g., making a general LLM fluent in healthcare or finance jargon). 👉 Adapters: Small bottleneck layers inserted between transformer blocks. The are quick to switch out and this makes it possible to maintain multiple domain-specialized variants efficiently. 👉 Prefix-Tuning: Prepends learned “soft prompts” to the model’s inputs, teaching it how to behave differently without changing any internal weights. It’s also lightweight and ideal for few-shot or multitask scenarios. Takeaway ✅ Pretraining helps LLM builds general intelligence and Fine-tuning makes the model useful for your task. ✅ PEFT makes it efficient to fine-tune LLMs. LoRA for performance and adapters/prefix-finetuning for lightweight multi-task settings. Tune in tomorrow for more SLM/LLMs deep dives. -- 🚶➡️ To learn more about LLMs/SLMs, follow me - Karun! ♻️ Share so others can learn, and you can build your LinkedIn presence! (Img Src: https://lnkd.in/gUEG4_nM)

𝗠𝗮𝘀𝘁𝗲𝗿𝗶𝗻𝗴 𝗟𝗟𝗠 𝗢𝗽𝘁𝗶𝗺𝗶𝘇𝗮𝘁𝗶𝗼𝗻 Large language models have transformed from simple text generators into intelligent reasoning systems powering search engines, enterprise copilots, and autonomous agents. Yet their accuracy, relevance, and efficiency depend on how we optimize them. There are three core techniques shaping this next wave of AI innovation: Context Engineering, Prompt Engineering, and Fine-Tuning. Each plays a distinct role, and the future belongs to those who know how to combine them effectively. 𝗖𝗼𝗻𝘁𝗲𝘅𝘁 𝗘𝗻𝗴𝗶𝗻𝗲𝗲𝗿𝗶𝗻𝗴 𝗚𝗼𝗮𝗹: Dynamically feed the model the right information at the right time without retraining. 𝗛𝗼𝘄 𝗶𝘁 𝘄𝗼𝗿𝗸𝘀: Chunk and embed documents, store them in vector databases such as Pinecone, Weaviate, FAISS, or Milvus, and retrieve the most relevant content using retrieval augmented generation. Tools like LangChain and LlamaIndex orchestrate this process, ensuring token efficiency and building dynamic contexts. 𝗨𝘀𝗲 𝗰𝗮𝘀𝗲: Enterprise knowledge assistants that instantly retrieve policies, Jira tickets, or AWS configurations on demand. 𝗣𝗿𝗼𝗺𝗽𝘁 𝗘𝗻𝗴𝗶𝗻𝗲𝗲𝗿𝗶𝗻𝗴 𝗚𝗼𝗮𝗹: Design high-quality prompts that maximize clarity, control, and reasoning depth. 𝗛𝗼𝘄 𝗶𝘁 𝘄𝗼𝗿𝗸𝘀: Define objectives, structure zero-shot or few-shot examples, leverage chain-of-thought reasoning, and continuously refine outputs through iterative testing and feedback loops. Tools such as OpenAI Playground, LangSmith, PromptFlow, and Weights & Biases make experimentation and evaluation seamless. 𝗨𝘀𝗲 𝗰𝗮𝘀𝗲: AI compliance reporting agents where precision and regulatory alignment are critical. 𝗙𝗶𝗻𝗲-𝗧𝘂𝗻𝗶𝗻𝗴 𝗚𝗼𝗮𝗹: Permanently teach an LLM domain-specific knowledge or custom behavior. 𝗛𝗼𝘄 𝗶𝘁 𝘄𝗼𝗿𝗸𝘀: Prepare high-quality labeled datasets, initialize a base model, and train using OpenAI Fine-Tuning API, Hugging Face Transformers, LoRA adapters, or AWS Sagemaker. Fine-tuning improves consistency and enables models to learn proprietary information and unique writing styles. 𝗨𝘀𝗲 𝗰𝗮𝘀𝗲: Training a medical AI assistant with proprietary datasets to improve diagnostic accuracy and decision support. 𝗧𝗵𝗲 𝗙𝘂𝘁𝘂𝗿𝗲 𝗼𝗳 𝗟𝗟𝗠 𝗘𝗻𝗴𝗶𝗻𝗲𝗲𝗿𝗶𝗻𝗴 Prompt engineering guides behavior. Context engineering supplies knowledge. Fine-tuning builds expertise. When combined, these disciplines enable engineers to design scalable, explainable, and production ready AI systems. Follow Umair Ahmad for more insights. #AI #LLM #ContextEngineering #PromptEngineering #FineTuning #MachineLearning #SystemDesign

I'm kicking off a series of posts here that I hope will gain some traction. Post 1: How to teach an LLM hard things without it forgetting how to tie its shoes Over the past 3 weeks, AI21 Labs released several posts under "Labs In Front"—a fascinating look into advanced but intuitive details about LLM training from some of the best AI researchers in Israel. Some background: A while back, DeepSeek introduced GRPO—a method for fine-tuning LLMs by having them generate multiple answers and automatically scoring which are correct. No human feedback needed. It works especially well for verifiable tasks (math, coding), but surprisingly improves models across the board. It's now industry standard. The problem: As training progresses, most tasks become too easy—the model outputs 10 answers, all correct, learns nothing. Or tasks are impossible (like RAG questions with no answer in the documents). Either way: zero learning signal. The researchers found that up to 85% of training outputs become useless. The naive fix: Stop showing examples the model already gets right. But this causes "starvation"—the model sees only hard examples and starts forgetting the easy stuff. Training becomes unstable. Even worse when learning multiple tasks: it keeps grinding on hard problems while forgetting how to tie its shoes. The solution: Brilliantly simple. Set an "alarm clock." If the model aces an example, snooze it for a while—but periodically wake it up and ask: "Hey, you still remember how to tie shoes? No? Let's refresh." This lets the model tackle increasingly difficult problems without catastrophic forgetting. Result: 3X training efficiency—which translates to serious cost savings. Link to the full post in comments Daniel Gissin, Yuval Globerson Inbal Magar Yaniv Markovski Sharon Argov