Brent Wei Liu

Get faster and smarter MOE inference straight out of the box. 👇Deep dive on scaling expert parallelism with TensorRT-LLM. LLMs with MOE promise higher model capacity without linearly increasing compute costs - but they introduce new challenges -- more conditional computation, dynamic routing, and non-uniform GPU utilization -- solved by TensorRT-LLM. ✨ New in TensorRT-LLM has native support for expert parallelism—designed for fast, efficient inference with MoE models like Mixtral (Mistral AI) and DeepSeek (DeepSeek AI). This gives you: ✅ Dynamic expert routing: Automatically route tokens to the top-k experts with minimal overhead. ✅ Efficient expert scheduling: Balance expert loads across GPUs using smart sharding and token bucketization. ✅ Memory-aware execution: Maximize hardware utilization while respecting memory budgets. ✅ Drop-in support: Use @HuggingFace models with minimal code changes via TensorRT-LLM's #Python API. 🧠 How it works: MoE models activate only a subset of "experts" for each token. This dynamic nature is powerful—but hard to optimize. It’s all done under the hood using custom #CUDA kernels and NCCL-based communication primitives—giving you low latency, high throughput, and better GPU scaling. ✨ TensorRT-LLM handles: ✅Token-expert mapping using the gating network. ✅Token sorting to batch same-expert tokens together. ✅Expert parallel execution across GPUs. ✅Merging outputs for final predictions. 🛠️ Developer workflow - here is the code to get started. # Clone the repo git clone https://lnkd.in/g-GiDX23 # Use included examples to load and run a Mixtral model cd TensorRT-LLM/examples/mixtral From there, the Python API lets you load the model, convert it with TensorRT, and run expert parallel inference—all with a few lines of code. Results? 📈 Performance at scale. Tests show up to 2.3x faster inference throughput compared to standard tensor parallelism when using 8 GPUs and top-2 experts per token. Even better—TensorRT-LLM keeps efficiency high across increasing batch sizes. Want to see it in action or contribute? 👉 Read the full tech blog: https://lnkd.in/g_7YV3vV 👉 Explore the code on GitHub: https://lnkd.in/gNjQ5W2U 👉 Follow updates in the TensorRT-LLM repo: https://lnkd.in/gqSHYQ4u Share your experiences with us.

78

3 Comments

Streamline your machine learning workflows with CI/CD in Julia. From model validation to deployment, discover how to build scalable, test-driven, and reproducible ML systems using Julia’s powerful ecosystem. https://lnkd.in/esWGw8MT #JuliaLang #MLOps #CI_CD #MachineLearning #programming

17

🧠💬 Memory in LLMs A practical guide showing how to implement conversational memory in LLMs using LangGraph, demonstrated through a therapy chatbot. Features code examples for basic retention, trimming, and summarization approaches. Learn to build memory-aware apps 👉 https://lnkd.in/gybcrV5v

967

21 Comments

New short course: DSPy: Build and Optimize Agentic Apps DSPy is a powerful open-source framework for automatically tuning prompts for GenAI applications. In this course, you'll learn to use DSPy, together with MLflow. This is built in partnership with Databricks and taught by Chen Qian, co-lead of the DSPy framework. Many AI builders spend hours hand-tuning prompts. When given a set of evals, DSPy automates this process. It’s especially useful for optimizing prompts, including few-shot prompts, in complex agentic AI workflows. Further, if you switch an application to a newer LLM, performance can degrade if your prompts were optimized to the previous model. DSPy automatically optimizes the entire system for the new LLM as well, using just a few evaluation examples. This course teaches DSPy works, and best practices for using it. You’ll write programs using DSPy’s signature-based programming model, debug them with MLflow tracing -- to gain visibility into how different parts of a pipeline, as well as how the overall system, are performing -- and automatically improve their accuracy with DSPy Optimizer. Please sign up here: https://lnkd.in/gdjae8AX

4,672

190 Comments

💡 What we’ve learned from Yingjun Wu at our latest Data Engineering Meetup Stream processing has always chased the holy grail of ultra-low latency at scale. Traditionally, that meant leaning on local storage engines. But in the cloud, things get tricky. Yingjun Wu, Founder & CEO of RisingWave, addressed this topic in his talk, which you can re-watch on YouTube: https://lnkd.in/eXUMSUBF 
For a quick overview, we collected some of our key takeaways for you: 🤷‍♀️ Not all clouds are equal Local storage is very fast, but it doesn’t scale easily. Cloud storage like Amazon S3 can hold huge amounts of data safely, but it’s much slower to access. RisingWave’s solution is Hummock, a system that stores frequently used data in memory, medium-use data on disk, and uses S3 for long-term storage, meaning queries never access S3 directly. 🦾 Compaction: Let the workers do the heavy lifting! 
Normally, query nodes also reorganize and clean up data (= compaction), which slows things down. RisingWave moves this to separate, so-called “remote workers,” keeping queries fast and smooth. ⏩ Low latency is possible even when scaling in the cloud By combining smart caching with an automatic data cleanup, consistently fast query times can be achieved. The system can scale up or down as the cloud allows, without slowing down due to data management. Proving: elasticity and low latency can coexist! → In summary: Stream processing no longer has to choose between speed, reliability, and scalability. But you need to rethink the old assumptions from single-machine designs when moving to the cloud.

18

1 Comment

We’ve worked with Heroku to make Pydantic AI integrate seamlessly with Heroku Managed Inference & Agents. This combination lets you: * Build agents in a familiar Pythonic style, with Pydantic’s data validation at the core * Deploy and run them on Heroku’s fully managed infrastructure * Use standard protocols like Model Context Protocol (MCP) and Agent-to-Agent (A2A) for interoperability The blog post walks through creating and deploying a Pydantic AI agent on Heroku in just a few steps. Read the guide here: https://lnkd.in/epwR98kH

67

2 Comments

Large Language Models can't access your private data unless you help them do so. In this tutorial, Maham shows you how to use Langbase's Memory Agents for this. You'll learn how to securely let LLMs chat with your docs by creating and integrating your own memory agent. https://lnkd.in/gZkYWyCw

387

4 Comments