Boxin Jiang

Get faster and smarter MOE inference straight out of the box. 👇Deep dive on scaling expert parallelism with TensorRT-LLM. LLMs with MOE promise higher model capacity without linearly increasing compute costs - but they introduce new challenges -- more conditional computation, dynamic routing, and non-uniform GPU utilization -- solved by TensorRT-LLM. ✨ New in TensorRT-LLM has native support for expert parallelism—designed for fast, efficient inference with MoE models like Mixtral (Mistral AI) and DeepSeek (DeepSeek AI). This gives you: ✅ Dynamic expert routing: Automatically route tokens to the top-k experts with minimal overhead. ✅ Efficient expert scheduling: Balance expert loads across GPUs using smart sharding and token bucketization. ✅ Memory-aware execution: Maximize hardware utilization while respecting memory budgets. ✅ Drop-in support: Use @HuggingFace models with minimal code changes via TensorRT-LLM's #Python API. 🧠 How it works: MoE models activate only a subset of "experts" for each token. This dynamic nature is powerful—but hard to optimize. It’s all done under the hood using custom #CUDA kernels and NCCL-based communication primitives—giving you low latency, high throughput, and better GPU scaling. ✨ TensorRT-LLM handles: ✅Token-expert mapping using the gating network. ✅Token sorting to batch same-expert tokens together. ✅Expert parallel execution across GPUs. ✅Merging outputs for final predictions. 🛠️ Developer workflow - here is the code to get started. # Clone the repo git clone https://lnkd.in/g-GiDX23 # Use included examples to load and run a Mixtral model cd TensorRT-LLM/examples/mixtral From there, the Python API lets you load the model, convert it with TensorRT, and run expert parallel inference—all with a few lines of code. Results? 📈 Performance at scale. Tests show up to 2.3x faster inference throughput compared to standard tensor parallelism when using 8 GPUs and top-2 experts per token. Even better—TensorRT-LLM keeps efficiency high across increasing batch sizes. Want to see it in action or contribute? 👉 Read the full tech blog: https://lnkd.in/g_7YV3vV 👉 Explore the code on GitHub: https://lnkd.in/gNjQ5W2U 👉 Follow updates in the TensorRT-LLM repo: https://lnkd.in/gqSHYQ4u Share your experiences with us.

78

3 Comments

Databricks Data Engineer with DevOps Skills Los Angeles CA (Hybrid) Long term No H1Bs&GCs     Must Demonstrate (Critical Skills & Architectural Competencies) Designing and implementing Databricks-based Lakehouse architectures on AWS Clear separation of compute vs. serving layers Ability to design low-latency data/API access strategies (beyond Spark-only patterns) Strong understanding of caching strategies for performance and cost optimization Data partitioning, storage optimization, and file layout strategy Ability to handle multi-terabyte structured or time-series datasets Skill in requirement probing, identifying what matters architecturally  12+ years of experience Strong hands-on experience with Databricks, including: Delta Lake Unity Catalog Lakehouse Architecture Delta Live Pipelines Databricks Runtime Table Triggers Databricks Workflows Proficiency in PySpark, Spark, and advanced SQL. Expertise with AWS cloud services, including: S3 IAM Glue / Glue Catalog Lambda Kinesis (optional but beneficial) Secrets Manager Strong understanding of DevOps tools: Git / GitLab CI/CD pipelines Databricks Asset Bundles Familiarity with Terraform is a plus. Experience with relational databases and data warehouse concepts. Please share suitable profiles to syam@s2jsolutions.com

We just published our first technical research note. "Low-Latency Event-Driven Architecture for Real-Time Options Execution on IBKR" It covers how we eliminated compounding feed lag at SPY market open — from a growing 45-second backlog to stable sub-second bar sealing — through: → Queue-first callback offloading (Python GIL problem) → Per-venue DOM isolation across ARCA, ISLAND, IEX → Event-time binning across 6 independent intervals → Single source of truth for all technical indicators No signals. No alpha. Pure infrastructure. This is the kind of work that happens before a single trade fires — and the kind that determines whether your system survives the opening bell. 📄 Download: https://lnkd.in/g-XJsYwG #AlgoTrading #OptionsTrading #MarketMicrostructure #SystematicTrading #Python #QuantitativeFinance #TradingInfrastructure

1

It’s been the headline of the week: 𝐈𝐁𝐌 𝐢𝐬 𝐚𝐜𝐪𝐮𝐢𝐫𝐢𝐧𝐠 𝐂𝐨𝐧𝐟𝐥𝐮𝐞𝐧𝐭. A big question on everyone’s mind is how this will reshape the streaming landscape. In Yingjun Wu’s words, this acquisition does not mark the end of anything! It only accelerates a shift that was already in motion. The whole data infra stack is moving upward, away from raw plumbing and toward products that deliver visible value. This shift shows up in three places: ▪️ How analytics vendors treat ingestion ▪️ How Apache Iceberg changes the role of Kafka ▪️ How Kafka itself has to evolve. For details, read: https://lnkd.in/gXUMRyPA #risingwave #streamprocessing #datainfra

5

We at Haladir (YC W26) released a report on RLFR: Reinforcement Learning from Formally-Defined Rewards for Code Generation (https://lnkd.in/guCH7w7C). The core idea is simple: we present a simple offline and single-turn RL framework that uses formal verification as the primary reward signal for program synthesis in LLMs rather than unit test-based rewards (RLVR), showing improvements over a swath of different coding benchmarks. We fine-tune both Qwen2.5-Coder-7B-Instruct and Qwen3-8B using RLFR and Dafny, a specification language that utilizes Boogie (IVL) and Z3 (SMT solver) to check verification conditions. Despite solely training on Dafny, we show improvements on Python-native coding benchmarks, suggesting generalization past the formal verification domain.

90

8 Comments

We (w/ Shansan Gong, Ruixiang ZHANG, Huangjie Zheng, Jiatao Gu, Navdeep Jaitly, Lingpeng Kong) released a family of 7B diffusion language models, DiffuCoder, that specializes on code generation, with a focus on understanding and improving masked diffusion models. A core analysis of DiffuCoder is the autoregressiveness (AR-ness) score, a novel metric that quantifies the causal patterns in decoding, revealing how diffusion models break from strict left-to-right generation for more flexible, non-linear code planning. Recent advances in autoregressive (AR) models dominate code generation, but diffusion-based LLMs (dLLMs) like DiffuCoder offer a promising alternative, especially for complex programming tasks. DiffuCoder explores how these models decode differently—showing less global AR-ness in code tasks compared to math—and how temperature affects both token selection and generation order, unlike traditional AR models. We also introduce coupled-GRPO, a post-training RL method with a coupled-sampling scheme, to reduce performance drops during accelerated decoding, boosting parallelism and efficiency. We use a self-improvement pipeline that leverages AR-ness analysis, coupled-GRPO optimization, and evaluation on benchmarks like AceCode-89k to refine decoding strategies. This approach enables DiffuCoder to navigate diverse code generation pathways and enhance performance with modest computational overhead. Looking ahead, we aim to further leverage Reinforcement Learning to steer code generation through these decoding patterns, with the discrete nature of AR-ness scores providing a foundation for search-based strategies—ideal for the sparse rewards of optimizing complex code structures. Check out our full paper and code for a deeper dive! Paper: https://lnkd.in/gVWU3BDJ Code: https://lnkd.in/gmXTZ_6n Models: https://lnkd.in/gTcKCDr9 #MachineLearning #AI #CodeGeneration #DiffusionModels #NLP

220

5 Comments

New research from Databricks. It's about training enterprise search agents via RL. KARL introduces a multi-task RL approach where agents are trained across heterogeneous search behaviors, constraint-driven entity search, cross-document synthesis, and tabular reasoning. It generalizes substantially better than those optimized for any single benchmark. KARL is Pareto-optimal on both cost-quality and latency-quality trade-offs compared to Claude 4.6 and GPT 5.2. With sufficient test-time compute, it surpasses the strongest closed models while being more cost efficient.

396

8 Comments

Sharing here Meta's work on torchcomms as a new PyTorch comms APIs and the open-sourcing of our NCCLX/CTran implementation that got us to scale to 100K+ GPUs last year and is taking us to gigawatt-scale. If you're at the PyTorch Conference today, you can attend a talk from the team this morning: https://lnkd.in/g25BauUV So proud of our Meta teams that worked on this! Coupled with our announcement from last week at OCP summit on our fabrics, switches, and scale-up (https://lnkd.in/g-33_3CZ), this shows Meta's continued drive for open networking for AI up and down the stack to support infra builders and model researchers. From the low-level networking hardware up to the comms software stack in PyTorch, there's an amazing amount of exciting work in networking for AI! #AIInfrastructure #Meta

41

We are reimagining PyTorch on TPU (https://lnkd.in/gCrM-TZD). This approach still calls XLA but in a profoundly different way, enabling a solid eager mode. In the limit, this new approach will enable a true JIT approach with recompilation and re-optimization, online feedback-directed optimizations, avoidance of OOMs, support for HW fungibility, hyper-optimizations of kernels, and more - all with close to zero code changes!

341

2 Comments

AI agents are only as powerful as their memory. Redis has been named the #1 data storage choice for AI agents in 2025. The reason? Redis delivers what AI truly needs: - Real-time vector storage - Semantic caching - Long-term memory for LLMs At Bassirah, we see this as validation of why Redis is central to the next generation of AI-native data platforms powering everything from agentic RAG systems to enterprise decision engines. 🔗 Read more: https://lnkd.in/db47k6zG #AI #Redis #RAG #LLM #DataEngineering #AIAgents #ArtificialIntelligence

2