Large Language Models Insights

For the last couple of years, Large Language Models (LLMs) have dominated AI, driving advancements in text generation, search, and automation. But 2025 marks a shift—one that moves beyond token-based predictions to a deeper, more structured understanding of language.  Meta’s Large Concept Models (LCMs), launched in December 2024, redefine AI’s ability to reason, generate, and interact by focusing on concepts rather than individual words.  Unlike LLMs, which rely on token-by-token generation, LCMs operate at a higher abstraction level, processing entire sentences and ideas as unified concepts. This shift enables AI to grasp deeper meaning, maintain coherence over longer contexts, and produce more structured outputs.  Attached is a fantastic graphic created by Manthan Patel How LCMs Work:  🔹 Conceptual Processing – Instead of breaking sentences into discrete words, LCMs encode entire ideas, allowing for higher-level reasoning and contextual depth.  🔹 SONAR Embeddings – A breakthrough in representation learning, SONAR embeddings capture the essence of a sentence rather than just its words, making AI more context-aware and language-agnostic.  🔹 Diffusion Techniques – Borrowing from the success of generative diffusion models, LCMs stabilize text generation, reducing hallucinations and improving reliability.  🔹 Quantization Methods – By refining how AI processes variations in input, LCMs improve robustness and minimize errors from small perturbations in phrasing.  🔹 Multimodal Integration – Unlike traditional LLMs that primarily process text, LCMs seamlessly integrate text, speech, and other data types, enabling more intuitive, cross-lingual AI interactions.  Why LCMs Are a Paradigm Shift:  ✔️ Deeper Understanding: LCMs go beyond word prediction to grasp the underlying intent and meaning behind a sentence.  ✔️ More Structured Outputs: Instead of just generating fluent text, LCMs organize thoughts logically, making them more useful for technical documentation, legal analysis, and complex reports.  ✔️ Improved Reasoning & Coherence: LLMs often lose track of long-range dependencies in text. LCMs, by processing entire ideas, maintain context better across long conversations and documents.  ✔️ Cross-Domain Applications: From research and enterprise AI to multilingual customer interactions, LCMs unlock new possibilities where traditional LLMs struggle.  LCMs vs. LLMs: The Key Differences  🔹 LLMs predict text at the token level, often leading to word-by-word optimizations rather than holistic comprehension.  🔹 LCMs process entire concepts, allowing for abstract reasoning and structured thought representation.  🔹 LLMs may struggle with context loss in long texts, while LCMs excel in maintaining coherence across extended interactions.  🔹 LCMs are more resistant to adversarial input variations, making them more reliable in critical applications like legal tech, enterprise AI, and scientific research.  

If you’re an AI engineer working on RAG, or building advanced retrieval-augmented systems, you need to know about RAFT: Retrieval-Augmented Fine-Tuning. Let’s break it down 👇 → Closed-Book Models (SFT Only) The model learns everything at train time, and answers based purely on its internal weights. Fast, but brittle – hallucinations spike when the model faces unfamiliar queries. → Open-Book Models (Standard RAG) At inference time, the model retrieves top-k documents and answers using them as context. But the model has never seen these docs during training – so it treats relevant and irrelevant documents the same way, often leading to noisy outputs. → RAFT: Retrieval + Fine-Tuning Combined RAFT, proposed by UC Berkley, merges RAG and fine-tuning. During training, the model is explicitly taught how to use retrieved documents – rewarding it for grounding answers in the right document and ignoring distractors. Here’s how RAFT works: → Use a query → Pair it with a golden doc (the correct reference) → Add sampled negative docs (distractors) → Train the model to generate an answer that quotes only from the golden doc This makes the model retrieval-aware during generation – it learns to differentiate between helpful and irrelevant documents. Why RAFT matters 🤔 → Reduces hallucinations by grounding answers in relevant context → Boosts accuracy in domain-specific applications like legal, medical, scientific QA → Works with smaller open-weight models like LLaMA 2 and Mistral 7B → Outperforms vanilla RAG on benchmarks like HotpotQA and PubMedQA How to train with RAFT 🛠️ → Build training triples: (query, golden doc, distractor docs) → Use your existing retrieval setup and corpus → Fine-tune using LoRA or full SFT with these inputs → At inference, continue to use top-k retrieval – the model will now handle noise better When to use RAFT ⁉️ → When your application requires faithfulness and traceability (e.g., legal, healthcare) → When your retrieval corpus includes overlapping or ambiguous docs → When you want smaller models to reason better with external documents RAFT doesn’t replace retrieval – it enhances it by teaching the model how to reason over retrieved content. Instead of hoping your model figures it out at runtime, RAFT prepares it during training. If you’re working on GenAI systems or retrieval pipelines, this is one method you can’t afford to ignore. Arvind and I are doing a free RAG lightning session on 4th April. If you want to learn more about RAG, do join us: https://lnkd.in/gHFmmfR2

Let me add a bit context to the latest DeepSeek code release as I feel it was a bit bare bones. Mixture-of-Experts (MoE) is a simple extension of transformers which is rapidly establishing itself as be the go-to architecture for mid-to-large size LLM (20B-600B parameters). It modifies the feedforward block by duplicating them in several "experts" with a router at the entry sending each input token to either one or another expert and a gathering operation at the end of the MoE block to bring the sequence together before the attention block. This change in architecture allows one to increase the total size/capacity of an AI model without increasing the number of operations seen by each token thereby theoretically allowing for smarter models with the same compute requirements i.e. latency (the price is paid in memory usage) However, MoEs bring a number of new challenges: because they require more memory and are usually used for mid-to-large models, they often need to be parallelized on multiple GPUs and the communication need to be super efficient because it's right in the critical path. We wrote a long blog post recently on these topics so feel free to take a deeper look here: https://lnkd.in/dfvqTaD7 There are currently only a few codebases that allow you to train MoE, including: - DeepSpeed: https://lnkd.in/dXkzrCCv - MegatronLM: https://lnkd.in/dvar4GBu - Databricks/MosaicML LLM Foundry: https://lnkd.in/dpk2Tfzk DeepSeek recently trained a State-of-the-art MoE called DeepSeek-R1 which grabbed world-wide attention in part because of its performance but also because it was trained extremely efficiently (only costing an estimate $6M to train). The model was also very efficiently run in inference. In this latest release they are open-sourcing a crucial element of this stack, namely the communication/orchestration library for the MoE part of the model, including some state-of-the-art FP8 support (DeepSeek R1 was also the first very large SOTA model trained with FP8 low-precision support afaik) Here is the link to this new (and already viral) DeepSeek repo: https://lnkd.in/dYNp4ZEi This is very exciting for all the teams training large models but also using them in inference as many super smart efficiency trick are to be found in this new codebase. So congrats to the DeepSeek AI team for sharing to openly their knowledge with the whole community! For more details on how MoE parallelism fits in the whole model training setup, feel free to check our open-source Ultra-Scale Playbook released last week here which covers all this and even more: https://lnkd.in/eDC9TwSj

🚀 DeepSeek Just Dropped 3 Powerful Open-Source Releases – Here’s Why They Matter They’re rewriting the rulebook on efficient LLM training and deployment. Today, they open-sourced three incredibly small (yet powerful) repositories, each addressing a key bottleneck in large-scale AI infrastructure.👇 1️⃣ Profiling Data for AI Training Efficiency On the surface, this might not seem groundbreaking, but this dataset is a goldmine. It provides a real-world breakdown of how DeepSeek keeps GPUs fully utilized during training and inference, ensuring that every single compute cycle contributes to efficiency. ✅ Optimized scheduling = faster, cheaper AI training ✅ Helps teams visualize GPU workload distribution (viewable in Chrome tracing tools) ✅ A rare, transparent look into state-of-the-art AI scaling techniques I wish more open-source teams would release this kind of data, because training efficiency is the #1 challenge at massive scales. 2️⃣ Load Balancing for Mixture of Experts (MoE) Mixture of Experts (MoE) is a major reason why AI models can scale efficiently, but there’s always been one major problem: some GPUs get overloaded while others sit idle. DeepSeek’s Expert Parallelism Load Balancer (EPLB) solves this by: ✅ Duplicating and redistributing heavyloaded experts across GPUs ✅ Minimizing internode traffic, reducing delays ✅ Ensuring balanced workloads, preventing bottlenecks This is huge! MoE models are notoriously tricky to optimize, and this tool simplifies deployment for anyone working with expert-based architectures. If you’re serious about scaling efficient MoE models, this is an absolute must-try. 3️⃣ The Game-Changer: DualPipe – Zero-Bubble Parallelism 🔥 This is THE most exciting part of today’s release. Pipeline Parallelism (PP) is used to split LLM training across GPUs, but it comes with inefficiencies—idle time (bubbles) between forward and backward passes. DualPipe eliminates these bubbles, achieving a “zero-bubble regime” for the first time ever in large-scale AI training. 💡 Why this is huge? - Full computation-communication overlap (no wasted cycles) - Reduces training time and cost significantly - First-of-its-kind implementation, never reported before in SOTA training If you work with distributed AI training, this could dramatically improve efficiency and lower costs across the board. Final Thoughts DeepSeek is doing open-source right. Instead of just releasing models, they’re sharing the critical tools and techniques that power SOTA AI training. - GPU efficiency matters, profiling data like this is rare and invaluable. - Mixture of Experts isn’t magic, it needs proper balancing. EPLB makes it easy. - Zero-bubble training is a reality. DualPipe might become the new standard! How do you see AI training evolving? links in the comments.

When you hear the phrase 'data distribution', your first instinct might be to tune out. But if you’re serious about shaping your organisations AI strategy, it’s one of the most important ideas to comprehend. Large language models like ChatGPT and Gemini learn by consuming oceans of text and images. Hidden within those vast datasets are patterns - the world’s concepts, relationships, and associations. The training data distribution describes how all those examples are spread out and connected. Inside that space, models perform astonishingly well: they compress knowledge into abstract patterns that let them talk confidently about everything from quantum mechanics to cooking recipes. But the edges of that distribution are treacherous. Feed a model something truly unfamiliar - something it’s never seen before - and performance doesn’t just decline, it collapses. It’s almost like triggering an adversarial attack: what was superhuman suddenly becomes sub–five-year-old. The model simply doesn’t know what it doesn’t know. That’s the first key insight. Models can still make dangerous mistakes in your area of expertise. The second insight is equally striking: everything inside the general training distribution has become cheap. Tasks and information that live within it - public facts, translation, summarisation, boilerplate code - are now commodities. Competing there is a race to zero. So the strategic question becomes: what does your organisation know that lies outside of the general distribution? What unique knowledge, experience, or data sits beyond what the models already contain? This realisation splits the playbook in two. On one side lies context, guardrails and resilience - internal systems that provide the missing context and recognise when a model has stepped out of its distribution. Think detection, retrieval grounding, context engineering, and oversight. These keep your agents safe and dependable. On the other side lies distinctiveness and value - identifying, structuring, and protecting the knowledge that only you possess. Every organisation has it: proprietary methods, tacit expertise, or specialised data. The challenge is organising it, connecting it together and then protecting it. That’s where knowledge graphs and ontologies step in. Ontologies capture meaning with mathematical precision; knowledge graphs connect that meaning to data. When integrated with your AI, they provide the guardrails that make models safer - and the distinctive context that makes them more accurate and valuable. The shared knowledge of humanity is being commoditised. What remains valuable is the uncommon - the structured, explainable, defensible edge of understanding that only you own. That’s why understanding data distribution matters. You need to know where the general distribution ends - so you can see where your advantage begins.

Most voice AI systems ignore 90% of the world’s languages. Why? Because data is scarce. Meta’s new Omnilingual Speech Recognition suite breaks that cycle. Existing models are trained on internet-rich languages and that dominates the research loop. Omnilingual can transcribe speech in over 1,600 languages, including 500 that no speech AI has ever supported. This is a glimpse into the next wave of AI: models that don’t assume the internet is the world. Highlights: – Transcription accuracy under 10% error for 78% of supported languages – In-context learning: adapt to new languages with just a few audio clips – Fully open-source: models, data, and the 7B Omnilingual w2v 2.0 foundation This isn’t about just recognizing speech. It’s about who gets included. If we can build models that work across dialects, cultures, and scarce data, the future of voice AI in enterprise, customer service, and global markets changes fast. - Announcement blog: https://go.meta.me/ff13fa - Download Omnilingual ASR: https://lnkd.in/g3w4FqY3 - Try the Language Exploration Demo: https://lnkd.in/gVzrcdbd - Try the Transcription Tool: https://lnkd.in/gRdZuZqP - Read the Paper: https://lnkd.in/giKrvniC

Want to boost LLM performance? Merge two LLMs together. I used to be active in data science competitions on Kaggle. The way to win a Kaggle competition is generally to create the biggest ensemble of models you can. Each model excels in its own corner of the prediction space, and when you put them together, you generally get a performance boost. Kind of like asking the same question of a lot of smart people. This same technique is coming to large language models. It is called merging. Merging is cost-effective (no GPU required) and produces winners. For example, the Marcoro14-7B-slerp model, created using the mergekit library (link below), became the best-performing model on the Open LLM Leaderboard as of Feb 1, 2024. The most common model merging technique is called SLERP (Spherical Linear Interpolation). Here’s how it works: 1/Normalization: The input vectors from the LLMs are normalized to unit length. This ensures they represent directions rather than magnitudes1. 2/Angle Calculation: The angle between these vectors is calculated using their dot product1. 3/Interpolation: Spherical Linear Interpolation (SLERP) is used to smoothly interpolate between the vectors1. It maintains a constant rate of change and preserves the geometric properties of the spherical space in which the vectors reside1. 4/Weight Calculation: Scale factors based on the interpolation factor and the angle between the vectors are computed. These factors are used to weigh the original vectors. 5/Vector Summation: The weighted vectors are then summed to obtain the interpolated vector. Another technique, BRANCH-SOLVE-MERGE (BSM) from Meta, has shown significant improvements in evaluation correctness and consistency for each LLM, enhancing human-LLM agreement by up to 26%, reducing length and pairwise position biases by up to 50%. It also improved the coherence of the stories while also improving constraint satisfaction by 12%. Want to try it out? Start with MergeKit (https://buff.ly/4bg4wU1) Here are a few more resources: BSM paper: https://buff.ly/3vn0uck LLM-Slerp-Merge: https://buff.ly/4a6bREH HuggingFace article on LLM merging: https://buff.ly/43s3hO1 #ArtificialIntelligence #AIResearch #DeepLearning #NLP #LLM #ModelMerging

Large Language Diffusion Models (LLaDA) Proposes a diffusion-based approach that can match or beat leading autoregressive LLMs in many tasks. If true, this could open a new path for large-scale language modeling beyond autoregression. More on the paper: Questioning autoregressive dominance While almost all large language models (LLMs) use the next-token prediction paradigm, the authors propose that key capabilities (scalability, in-context learning, instruction-following) actually derive from general generative principles rather than strictly from autoregressive modeling. Masked diffusion + Transformers LLaDA is built on a masked diffusion framework that learns by progressively masking tokens and training a Transformer to recover the original text. This yields a non-autoregressive generative model—potentially addressing left-to-right constraints in standard LLMs. Strong scalability Trained on 2.3T tokens (8B parameters), LLaDA performs competitively with top LLaMA-based LLMs across math (GSM8K, MATH), code (HumanEval), and general benchmarks (MMLU). It demonstrates that the diffusion paradigm scales similarly well to autoregressive baselines. Breaks the “reversal curse” LLaDA shows balanced forward/backward reasoning, outperforming GPT-4 and other AR models on reversal tasks (e.g. reversing a poem line). Because diffusion does not enforce left-to-right generation, it is robust at backward completions. Multi-turn dialogue and instruction-following After supervised fine-tuning, LLaDA can carry on multi-turn conversations. It exhibits strong instruction adherence and fluency similar to chat-based AR LLMs—further evidence that advanced LLM traits do not necessarily rely on autoregression. https://lnkd.in/eYp9Hi5y

Based on over 1,100 curated papers and announcements featured throughout the year - the AI Tidbits SOTA report for 2023 is out. Just before we yell at ChatGPT once again as it got one detail wrong, let’s review the state-of-the-art today compared to December 2022 across various generative AI verticals. https://lnkd.in/gkBykSdS Here's a glimpse from the report: (1) Language models - within a year, the open-source community welcomed models like Yi and Mistral's Mixture of Experts that outperformed GPT-3.5. Meanwhile, commercial models like GPT-4 and Claude 2.1 continued to push the boundaries of language understanding, achieving exceptional scores in medical and bar exams and placing them among the top percentile. (2) Multimodal AI - 2023 was a stellar year with models like CogVLM, LLaVA, and GPT-4V(ision) demonstrating an unparalleled ability to process and interpret multiple forms of data, bringing us closer to AI that mimics human sensory inputs. (3) Autonomous agents - we saw groundbreaking progress in autonomous agents frameworks like AutoGPT and open-source models like CogAgent, signaling a near future where AI companions are an integral part of our everyday lives. (4) Image generation - it’s hard to believe that image diffusion models as we know them are less than two years old. DALL-E 3 and Midjourney led the pack in 2023, elevating the art of image synthesis and making it more accessible through ChatGPT and packages like Fooocus. No more deformed hands and faces or non-readable text. That’s 2022. (5) Video generation - Pika Labs and Runway were at the forefront with their foundation models, significantly improving video duration and quality in 2023. Meta's release of Emu Video and open-source projects like VideoCrafter1 also made notable contributions to this rapidly evolving space. (6) Speech understanding and generation - OpenAI’s Whisper and Deepgram’s Nova-2 showcased remarkable improvements in transcription accuracy, while ElevenLabs' text-to-speech model blurred the line between AI-generated and human voices, supporting input streaming for real-time speech synthesis. (7) Music generation - Meta’s MusicGen and Suno AI transformed text and melodies into music, marking a new era in AI-powered customized music creation. 2023 was a year where generative AI not only matched but, in many cases, surpassed human capabilities across various modalities. The open-source community particularly shined, boasting nearly 1,000 models on Hugging Face's Open LLM Leaderboard. 2024 could be the year in which an open-source model (powered by Mistral's next release?) surpasses GPT, AI companions become part of our daily lives through on-device small language models, and people no longer believe what they cannot physically touch. For a deep dive into these developments and a comparison between the state-of-the-art in 2022 and 2023, check out the full AI Tidbits 2023 SOTA Report https://lnkd.in/gkBykSdS

Hallucination in large language models (LLMs) has been widely studied, but the key question remains: Can it ever be eliminated? A recent paper systematically dismantles the idea that hallucination can be fully eradicated. Instead, it argues that hallucination is not just an incidental flaw but an inherent limitation of LLMs. 1️⃣ Hallucination is Unavoidable The paper establishes that LLMs cannot learn all computable functions, meaning they will inevitably generate incorrect outputs. Even with perfect training data, LLMs cannot always produce factually correct responses due to inherent computational constraints. No matter how much we refine architectures, training data, or mitigation techniques, hallucination cannot be eliminated—only minimized. 2️⃣ Mathematical Proofs of Hallucination They use concepts from learning theory and diagonalization arguments to prove that any LLM will fail on certain inputs. The research outlines that LLMs, even in their most optimized state, will hallucinate on infinitely many inputs when faced with complex, computation-heavy problems. 3️⃣ Identifying Hallucination-Prone Tasks Certain problem types are guaranteed to trigger hallucinations due to their computational complexity: 🔹 NP-complete problems (e.g., Boolean satisfiability) 🔹 Presburger arithmetic (exponential complexity) 🔹 Logical reasoning and entailment (undecidable problems) This means that asking LLMs to reason about intricate logic or mathematical problems will often lead to errors. 4️⃣ Why More Data and Bigger Models Won’t Fix It A common assumption is that hallucination can be mitigated by scaling—adding more parameters or training data. The paper challenges this notion: While larger models improve accuracy, they do not eliminate hallucination for complex, unsolvable problems. 5️⃣ Mitigation Strategies and Their Limitations Various techniques have been introduced to reduce hallucinations, but none can completely eliminate them: ✅ Retrieval-Augmented Generation (RAG) – helps provide factual grounding but does not guarantee accuracy. ✅ Chain-of-Thought Prompting – improves reasoning but does not fix fundamental hallucination limits. ✅ Guardrails & External Tools – can reduce risk but require human oversight. They suggest LLMs should never be used for fully autonomous decision-making in safety-critical applications. The Bigger Question: How Do We Build Safe AI? If hallucination is an unavoidable reality of LLMs, how do we ensure safe deployment? The research makes it clear: LLMs should not be blindly trusted. They should be integrated into workflows with: 🔹 Human in the loop 🔹 External fact-checking systems 🔹 Strict guidelines Are we designing AI with realistic expectations, or are we setting ourselves up for failure by expecting perfection? Should LLMs be used in high-stakes environments despite their hallucinations, or should we rethink their applications? #ai #artificialintelligence #technology