Minghan (Chris) Xu

Get faster and smarter MOE inference straight out of the box. 👇Deep dive on scaling expert parallelism with TensorRT-LLM. LLMs with MOE promise higher model capacity without linearly increasing compute costs - but they introduce new challenges -- more conditional computation, dynamic routing, and non-uniform GPU utilization -- solved by TensorRT-LLM. ✨ New in TensorRT-LLM has native support for expert parallelism—designed for fast, efficient inference with MoE models like Mixtral (Mistral AI) and DeepSeek (DeepSeek AI). This gives you: ✅ Dynamic expert routing: Automatically route tokens to the top-k experts with minimal overhead. ✅ Efficient expert scheduling: Balance expert loads across GPUs using smart sharding and token bucketization. ✅ Memory-aware execution: Maximize hardware utilization while respecting memory budgets. ✅ Drop-in support: Use @HuggingFace models with minimal code changes via TensorRT-LLM's #Python API. 🧠 How it works: MoE models activate only a subset of "experts" for each token. This dynamic nature is powerful—but hard to optimize. It’s all done under the hood using custom #CUDA kernels and NCCL-based communication primitives—giving you low latency, high throughput, and better GPU scaling. ✨ TensorRT-LLM handles: ✅Token-expert mapping using the gating network. ✅Token sorting to batch same-expert tokens together. ✅Expert parallel execution across GPUs. ✅Merging outputs for final predictions. 🛠️ Developer workflow - here is the code to get started. # Clone the repo git clone https://lnkd.in/g-GiDX23 # Use included examples to load and run a Mixtral model cd TensorRT-LLM/examples/mixtral From there, the Python API lets you load the model, convert it with TensorRT, and run expert parallel inference—all with a few lines of code. Results? 📈 Performance at scale. Tests show up to 2.3x faster inference throughput compared to standard tensor parallelism when using 8 GPUs and top-2 experts per token. Even better—TensorRT-LLM keeps efficiency high across increasing batch sizes. Want to see it in action or contribute? 👉 Read the full tech blog: https://lnkd.in/g_7YV3vV 👉 Explore the code on GitHub: https://lnkd.in/gNjQ5W2U 👉 Follow updates in the TensorRT-LLM repo: https://lnkd.in/gqSHYQ4u Share your experiences with us.

78

3 Comments

Most existing #lean4 datasets contain only correct proofs. Models learn error correction with RL, that's expensive. With UW Math AI lab we release a dataset of 260k erroneous Lean proofs with - compiler feedback - reasoning trace - corrected proof Improvements in Error Correction: - Goedel 8B: 2x - Kimina 8B: 3x Paper: https://lnkd.in/gaybt4bd

117

2 Comments

We (w/ Shansan Gong, Ruixiang ZHANG, Huangjie Zheng, Jiatao Gu, Navdeep Jaitly, Lingpeng Kong) released a family of 7B diffusion language models, DiffuCoder, that specializes on code generation, with a focus on understanding and improving masked diffusion models. A core analysis of DiffuCoder is the autoregressiveness (AR-ness) score, a novel metric that quantifies the causal patterns in decoding, revealing how diffusion models break from strict left-to-right generation for more flexible, non-linear code planning. Recent advances in autoregressive (AR) models dominate code generation, but diffusion-based LLMs (dLLMs) like DiffuCoder offer a promising alternative, especially for complex programming tasks. DiffuCoder explores how these models decode differently—showing less global AR-ness in code tasks compared to math—and how temperature affects both token selection and generation order, unlike traditional AR models. We also introduce coupled-GRPO, a post-training RL method with a coupled-sampling scheme, to reduce performance drops during accelerated decoding, boosting parallelism and efficiency. We use a self-improvement pipeline that leverages AR-ness analysis, coupled-GRPO optimization, and evaluation on benchmarks like AceCode-89k to refine decoding strategies. This approach enables DiffuCoder to navigate diverse code generation pathways and enhance performance with modest computational overhead. Looking ahead, we aim to further leverage Reinforcement Learning to steer code generation through these decoding patterns, with the discrete nature of AR-ness scores providing a foundation for search-based strategies—ideal for the sparse rewards of optimizing complex code structures. Check out our full paper and code for a deeper dive! Paper: https://lnkd.in/gVWU3BDJ Code: https://lnkd.in/gmXTZ_6n Models: https://lnkd.in/gTcKCDr9 #MachineLearning #AI #CodeGeneration #DiffusionModels #NLP

220

5 Comments

We at Haladir (YC W26) released a report on RLFR: Reinforcement Learning from Formally-Defined Rewards for Code Generation (https://lnkd.in/guCH7w7C). The core idea is simple: we present a simple offline and single-turn RL framework that uses formal verification as the primary reward signal for program synthesis in LLMs rather than unit test-based rewards (RLVR), showing improvements over a swath of different coding benchmarks. We fine-tune both Qwen2.5-Coder-7B-Instruct and Qwen3-8B using RLFR and Dafny, a specification language that utilizes Boogie (IVL) and Z3 (SMT solver) to check verification conditions. Despite solely training on Dafny, we show improvements on Python-native coding benchmarks, suggesting generalization past the formal verification domain.

90

8 Comments

𝗧𝗵𝗲 𝗯𝗲𝘀𝘁 𝗔𝗜 𝗰𝗼𝗱𝗶𝗻𝗴 𝗹𝗲𝘀𝘀𝗼𝗻 𝗰𝗼𝘀𝘁 𝗼𝘂𝗿 𝗲𝗻𝗴𝗶𝗻𝗲𝗲𝗿 $𝟲𝟬𝟬 𝗮𝗻𝗱 𝗮 𝗺𝗮𝗿𝗿𝗶𝗮𝗴𝗲 𝗮𝗿𝗴𝘂𝗺𝗲𝗻𝘁. Our VP of Engineering, Xiaofan(James) Luan, was supposed to buy his wife a Dior bag for their anniversary.  Instead, he bought three Claude Code subscriptions and spent the holiday trying to cross-compile 2 million lines of C++. Every fix on one platform broke two others.  $600 later, the only output was "git reset --hard" — and a very cold dinner table.😂 "Make it compile on Windows" is a trap. The real goal was "compile everywhere without hacks" — no AI is going to figure that out for you at 2 am. What worked: constraints before code, review tests not code, bottom-up, one layer at a time. Same task, two days. Then he ran six parallel Claude sessions across three machines with git worktree. The bottleneck stopped being intelligence and started being how fast one person can alt-tab. AI solves exactly the problem you give it. Engineering is in knowing which one to give. His wife is still waiting for that bag. Full story: https://lnkd.in/gtsW_Wvk ——— Follow Milvus, created by Zilliz, for everything related to unstructured data

9

We developed an open-source AI tool to design concrete mixes that are stronger, more sustainable, and ready to build with faster—speeding up construction while reducing environmental impact. https://lnkd.in/gPCk8tCM But the impact of this AI tool is not just hypothetical! Amrize used Meta’s AI-based technologies to design a new low-carbon mix, and successfully deployed it in an at-scale slab-on-grade application at Meta's new data center in Rosemont, MN. Compared to the legacy mix, this new AI-designed mix is: 🦁 Stronger ⏱ Faster 🍃 Lower carbon ⏱ ️The ideal set time All this was achieved without needing any new materials, nor special equipment. Best of all, the AI is open-sourced. https://lnkd.in/g2KA7KZW This work was featured in a Meta engineering blog article published today! https://lnkd.in/gBU9HY8H

98

9 Comments

🚗 Pricing at Lyft – A Masterclass in Data-Driven Decision Making Just came across this insightful article on Lyft’s pricing system by Yanqiao Wang and it truly impressed me. It explains how Lyft uses reinforcement learning and causal inference to optimize prices in real time. The system constantly learns from market behavior to balance supply, demand, and reliability for both riders and drivers. What stood out most is how data science, AI, and operations work together to make pricing fair, transparent, and adaptive. It shows how advanced technology can turn a complex problem like ride pricing into a continuous feedback loop that improves the marketplace experience for everyone. If you are interested in AI-driven decision systems, real-time analytics, or marketplace optimization, this is a must-read: https://lnkd.in/gRjJpDHp Thanks to Venkata Naga Sai Kumar Bysani for sharing this gem and highlighting such powerful real-world data applications. 🤝 Would love to connect with others who are passionate about data, technology, and building intelligent systems. Always open to learning, sharing, and exploring new ideas together. 🚀 #DataScience #ArtificialIntelligence #MachineLearning #MarketplaceOptimization

29

2 Comments

🚀 Excited to share my latest project! I’ve implemented Qwen 3.5 (0.8B) completely from scratch in PyTorch. It aligns perfectly with the official HuggingFace checkpoint, but I designed this specifically to make the architecture easy to understand, tweak, and experiment with. Here is what makes this architecture stand out from a standard Transformer: 🧠 Hybrid Attention Design : • 24 layers structured as (3 Linear + 1 Full Attention) × 6. • GatedDeltaNet (Linear Attention): Uses the delta-rule with learned decay (similar to Mamba-style memory). • Grouped Query Attention (GQA): Efficient full attention with shared KV heads. ⚙️ Modern LLM Components : • Partial RoPE (applied to 25% of head dimensions). • SiLU-gated FFN (3584 hidden dim). • RMSNorm + gated variants. • Support for massive long context (262K tokens). 🛠️ Clean & Modular Implementation : • Standalone attention modules (GQA + GatedDeltaNet). • HF checkpoint to custom model weight mapping. • Included Tokenizer + sampling loop. • End-to-end inference script (automatically downloads weights). 🔗 Check out the code here: https://lnkd.in/dZ3YFM9E If you're exploring how models like Qwen, LLaMA, or Mistral actually work under the hood, this might be useful.

18

2 Comments

Fun fact: Some of the hardest math at Amazon isn’t about AI, it’s about trucks. Curious how we solve it? We built a solver, and today, it’s open-source. 🚀 👉 Check it out & drop us a ⭐: https://lnkd.in/epTCYN8a Meet 𝗝𝘂𝗟𝗦 (Julia Local Search), Amazon Science’s new open-source solver: Developed by scientists, for scientists, JuLS is fast, extensible, and open to community contributions. It combines Constraint-Based Local Search and Constraint Programming, and it’s already deployed in Amazon’s production environment to tackle complex middle-mile logistics and scheduling. JuLS performs on par with Google OR-Tools on standard tasks. Its real strength lies in handling black-box objectives and constraints, making it a powerful solution for real-world optimization scenarios that often require interfacing with proprietary or external systems. More than a solver, JuLS is a playground for researchers and practitioners tackling combinatorial optimization problems. Our team : Axel Navarro, Arthur Dupuis, Ilan Coulon, Maxime Mulamba Ke Tchomba, El Mehdi OUDAOUD

302

9 Comments

History is repeating itself. 10 years ago the recsys industry was migrating from linear model recsys to sparse neural network recsys. Back in 2016, many people in the field were skeptical that deep learning would deliver positive ROI for recsys, the fear being rooted in the perceived higher training and inference cost. I see people repeat the same fear, uncertainty, and doubt about this year's migration from sparse neural network recsys to LLM-native recsys. Those who do not lean into the LLM-recsys migration this year will fall behind.

78

1 Comment