Satya Tumati

Get faster and smarter MOE inference straight out of the box. 👇Deep dive on scaling expert parallelism with TensorRT-LLM. LLMs with MOE promise higher model capacity without linearly increasing compute costs - but they introduce new challenges -- more conditional computation, dynamic routing, and non-uniform GPU utilization -- solved by TensorRT-LLM. ✨ New in TensorRT-LLM has native support for expert parallelism—designed for fast, efficient inference with MoE models like Mixtral (Mistral AI) and DeepSeek (DeepSeek AI). This gives you: ✅ Dynamic expert routing: Automatically route tokens to the top-k experts with minimal overhead. ✅ Efficient expert scheduling: Balance expert loads across GPUs using smart sharding and token bucketization. ✅ Memory-aware execution: Maximize hardware utilization while respecting memory budgets. ✅ Drop-in support: Use @HuggingFace models with minimal code changes via TensorRT-LLM's #Python API. 🧠 How it works: MoE models activate only a subset of "experts" for each token. This dynamic nature is powerful—but hard to optimize. It’s all done under the hood using custom #CUDA kernels and NCCL-based communication primitives—giving you low latency, high throughput, and better GPU scaling. ✨ TensorRT-LLM handles: ✅Token-expert mapping using the gating network. ✅Token sorting to batch same-expert tokens together. ✅Expert parallel execution across GPUs. ✅Merging outputs for final predictions. 🛠️ Developer workflow - here is the code to get started. # Clone the repo git clone https://lnkd.in/g-GiDX23 # Use included examples to load and run a Mixtral model cd TensorRT-LLM/examples/mixtral From there, the Python API lets you load the model, convert it with TensorRT, and run expert parallel inference—all with a few lines of code. Results? 📈 Performance at scale. Tests show up to 2.3x faster inference throughput compared to standard tensor parallelism when using 8 GPUs and top-2 experts per token. Even better—TensorRT-LLM keeps efficiency high across increasing batch sizes. Want to see it in action or contribute? 👉 Read the full tech blog: https://lnkd.in/g_7YV3vV 👉 Explore the code on GitHub: https://lnkd.in/gNjQ5W2U 👉 Follow updates in the TensorRT-LLM repo: https://lnkd.in/gqSHYQ4u Share your experiences with us.

78

3 Comments

LLMs look impressive in demos — but become inconsistent once they're embedded in real workflows. BASE Studio helps teams fine-tune models from SFT to RLHF using data built for real-world use, not toy examples: 🔵 Multi-turn conversations that preserve context 🔵 Clear, followable instructions models can reliably execute 🔵 Code data that explains *why* decisions are made — not just the final answer 🔵 Reasoning data that strengthens step-by-step thinking For multimodal models, we also align prompts with visual inputs to reduce inconsistency in text-to-image and text-to-video generation. The result? Models that behave more consistently — even after deployment. Learn more 👉 https://boden.ai #EmbodiedAI #Robotics #AITraining #TrainingData #MachineLearning #DataPipeline #RobotLearning #MultimodalAI #Automation #ComputerVision #AIInfrastructure #Annotation

1

Tuning Spark, Ray, or Dask for ML isn’t one-size-fits-all. In a new Domino Blueprint, Principal Solution Architect Sameer Wadkar shows how API-driven scaling complements autoscaling to right-size clusters on demand—cutting idle spend and speeding bursts for GPU/I/O-heavy work. Practical steps, a lightweight Python client, and Helm install included. If you care about FinOps and MLOps, this is for you. Read the blueprint: https://lnkd.in/ga_qJkhC

7

LLMs are powerful, until they melt your GPU budget and slow everything down. Today, Hrittik Roy and Aditya Soni will break down how they turned an unpredictable, resource-hungry LLM deployment into a fast, efficient, production-ready stack using CNCF tools. They’ll walk through: - Serving LLMs reliably with KServe + Kubeflow - Auto-scaling with Volcano and KEDA - Tracking performance + drift with Prometheus, Grafana & OpenTelemetry If you’re running LLMs in production... or trying to, this is a session worth joining. 🕒 2:30–3:00 PM EST 📍 Thomas Murphy Ballroom 4 | Level 5 #KubeCon #CloudNativeCon #vCluster

14

I love seeing ML teams who refuse to treat biology as an afterthought. Relation ‘s PatchDNA work is one of the clearest examples this year of what happens when you start with how biology actually works and only then design the model. No shortcuts, no flattening of the problem just to fit the tooling. That mindset hits close to home for me. At Outpost Bio, we’re obsessed with the same question: how do you build AI that doesn’t attempt to overwrite biology, but explains it? When you’re dealing with trillions of microbes shaping what we absorb, how drugs behave, and how our bodies respond to the world (biology’s “hidden majority” that most research ignores), you can’t get away with convenient abstractions. You have to let the system speak for itself. PatchDNA is a great example of doing that well: a biologically grounded inductive bias for DNA that gives the model structure instead of asking it to guess. Huge congrats to the Relation + Rosalind teams on the NeurIPS Best Paper. This is the direction the field needs: ML that respects mechanism. Congratulations! Cristian Regep Alice Del Vecchio Lindsay Edwards Rosie Rodriguez PhD Jake P. Taylor-King Sara-Jane Dunn #NeurIPS2025 #NeurIPS 🏆 Best Paper Award: NeurIPS AI4D3 Workshop PatchDNA: A flexible and biologically informed alternative to tokenization for DNA https://lnkd.in/dPHswTGP 🎤 Talk: Saturday, December 6th, 11:30 Convention Center, Upper Level, Room 28A–E 🖼️ Poster Session: Sunday, December 7th, 15:55–17:00 AI4Science Workshop, Convention Center, Upper Level, Ballroom 20A

85

4 Comments

Quantization in Large Language Models (LLMs) As LLMs scale, intelligence is no longer the bottleneck — memory, latency, and cost are. Quantization turns out to be one of the most powerful (and misunderstood) techniques enabling real-world deployment of large models. In this blog, I break down: • Why LLMs are surprisingly quantization-friendly • INT8 vs INT4 vs NF4 — what actually works in practice • How modern architectures absorb low-bit noise • Where quantization fails and how to design around it Quantization is no longer an optimization step — it’s foundational infrastructure for production LLM systems. Link: https://lnkd.in/gqGje75G #LLMs #AIEngineering #Quantization #MachineLearning #GenAI #MLOps

9

2 Comments

Building a Large Language Model (LLM) isn’t just about massive compute or model architecture; it’s about the data journey that gets it there. At Cogito Tech, we support this journey end-to-end, from curating domain-specific datasets and aligning transformer-based architectures to fine-tuning, evaluation, and continuous improvement. Every stage — data collection, annotation, and human-in-the-loop validation enhances precision and context that take models from prototype to production, faster. Here’s a quick snapshot of how we help build smarter, robust and domain-aware LLMs. https://lnkd.in/ef6DtJra #HumanInTheLoop #FineTuning #LLM #AI #DataCollection #DataAnnotation #GenerativeAI #ModelTraining #CogitoTech

19

🧾 Manifold Constrained Hyper Connections by DeepSeek Thanks to Uday Kamath, Ph.D. - who has a fantastic write-up on this. ❓What is it? A new technique that enables training higher quality LLMs with minimal increases in overhead. Modern Transformers have a highway “fast lane” to remember the initial “query”. To date, this has been limited to a small number of lanes. Previously, Deep Seek had shown that increasing the number of lanes improves quality…but that came with a lot of risk. Large-scale trainings could explode out of control, at immense capital cost. By constraining the manifolds, they shift the efficient frontier and increase reward with minimum increase in risk. A big breakthrough. 📈 Why it matters to business • Foundation models can have sizable jumps in quality with minimal costs changes. • The largest models can now be more ambitious without an increased fear of failed trainings. 🛠️ The technology details • Residual connection lanes are widened to incorporate more features. The new residuals have learned pre and post transformations. • In order to achieve stability, they incorporated the 1960s Sinkhorn-Knopp algorithm, which transforms the residual connection into a doubly stochastic matrix. • This markedly increased stability of training, measured by the Amax Gain Magnitude. • The team also devised some interesting innovations in memory management and gradient checkpointing in order to achieve these results. Uday’s Writeup https://lnkd.in/eSD8i9wP Paper https://lnkd.in/eWAgV_Sa

13