Max K.

Get faster and smarter MOE inference straight out of the box. 👇Deep dive on scaling expert parallelism with TensorRT-LLM. LLMs with MOE promise higher model capacity without linearly increasing compute costs - but they introduce new challenges -- more conditional computation, dynamic routing, and non-uniform GPU utilization -- solved by TensorRT-LLM. ✨ New in TensorRT-LLM has native support for expert parallelism—designed for fast, efficient inference with MoE models like Mixtral (Mistral AI) and DeepSeek (DeepSeek AI). This gives you: ✅ Dynamic expert routing: Automatically route tokens to the top-k experts with minimal overhead. ✅ Efficient expert scheduling: Balance expert loads across GPUs using smart sharding and token bucketization. ✅ Memory-aware execution: Maximize hardware utilization while respecting memory budgets. ✅ Drop-in support: Use @HuggingFace models with minimal code changes via TensorRT-LLM's #Python API. 🧠 How it works: MoE models activate only a subset of "experts" for each token. This dynamic nature is powerful—but hard to optimize. It’s all done under the hood using custom #CUDA kernels and NCCL-based communication primitives—giving you low latency, high throughput, and better GPU scaling. ✨ TensorRT-LLM handles: ✅Token-expert mapping using the gating network. ✅Token sorting to batch same-expert tokens together. ✅Expert parallel execution across GPUs. ✅Merging outputs for final predictions. 🛠️ Developer workflow - here is the code to get started. # Clone the repo git clone https://lnkd.in/g-GiDX23 # Use included examples to load and run a Mixtral model cd TensorRT-LLM/examples/mixtral From there, the Python API lets you load the model, convert it with TensorRT, and run expert parallel inference—all with a few lines of code. Results? 📈 Performance at scale. Tests show up to 2.3x faster inference throughput compared to standard tensor parallelism when using 8 GPUs and top-2 experts per token. Even better—TensorRT-LLM keeps efficiency high across increasing batch sizes. Want to see it in action or contribute? 👉 Read the full tech blog: https://lnkd.in/g_7YV3vV 👉 Explore the code on GitHub: https://lnkd.in/gNjQ5W2U 👉 Follow updates in the TensorRT-LLM repo: https://lnkd.in/gqSHYQ4u Share your experiences with us.

78

3 Comments

We developed an open-source AI tool to design concrete mixes that are stronger, more sustainable, and ready to build with faster—speeding up construction while reducing environmental impact. https://lnkd.in/gPCk8tCM But the impact of this AI tool is not just hypothetical! Amrize used Meta’s AI-based technologies to design a new low-carbon mix, and successfully deployed it in an at-scale slab-on-grade application at Meta's new data center in Rosemont, MN. Compared to the legacy mix, this new AI-designed mix is: 🦁 Stronger ⏱ Faster 🍃 Lower carbon ⏱ ️The ideal set time All this was achieved without needing any new materials, nor special equipment. Best of all, the AI is open-sourced. https://lnkd.in/g2KA7KZW This work was featured in a Meta engineering blog article published today! https://lnkd.in/gBU9HY8H

98

9 Comments

NVIDIA TensorRT LLM provides a high-level Python LLM API, with its PyTorch-native architecture enabling developers to experiment with the runtime or extend functionality. Learn how these latest TensorRT-LLM optimizations boost reasoning inference performance in our recent blog post. 🖇️ https://lnkd.in/gx52zwTA #PyTorch #OpenSourceAI #AI #Inference #Innovation

94

2 Comments

Two regimes. Two platforms. Two different shapes. I've been testing my Two State Ontology (TSO) framework on quantum hardware — IBM Marrakesh (superconducting qubits) and Quandela (photonic). TSO predicts that the shape of decoherence depends on whether a system is being actively driven through a critical interaction threshold (Γ_c). Below Γ_c (idle qubits, no interaction): standard exponential decay. At Γ_c (driven dynamics, active coupling): sigmoid/tanh transition. This week's results: • IBM Marrakesh idle decay: Both superposition (X2) and entanglement (X1) circuits decay exponentially. ΔAIC > 30 — decisive. This is what TSO predicts when no path rotation is occurring. • Photonic bunching cascade (Quandela SLOS): tanh preferred over exponential by ΔAIC > 25. Measured κ = 1.325 against TSO's prediction of 4/3 = 1.333 — a 0.6% deviation. • Kim et al. Rydberg re-analysis (129,791 shots): tanh beats exponential 2.75×. The pattern: standard QM below the threshold, TSO-specific behavior at it. Same relationship statistical mechanics has with thermodynamics — ordinary behavior in most regimes, critical phenomena only at phase transitions. TSO has zero free parameters. All constants derived from established physics. The decisive test remains: Rydberg atom arrays with tunable interaction strength through Γ_c, where TSO predicts ~35% deviation from exponential. All notebooks public. All failures documented (including 3+ killed versions). The framework evolved from first principles before any hardware testing. https://incapp.org/TSO/ #QuantumComputing #Physics #QuantumFoundations #OpenScience

Read Chaim Rand's practical walkthrough for optimizing PyTorch inference. Learn about batched inference, memory formats, IPEX, and graph compilation on Intel Corporation Xeon.

5

Transformer attention layers operate by projecting high-dimensional token representations into low-rank subspaces defined by per-head key/query/value projections. This inherently places a rank constraint on how relationships between items can be represented, and it forces the model to approximate rich interactions via combinations of rank-limited components. Our paper “Orthogonalization of data via Gromov-Wasserstein type feedback for clustering and visualization” provides a theoretical and algorithmic framework for dealing with full and low-rank structure in relational data representations by adapting transition probabilities and iteratively refining cluster orthogonality via Gromov-Wasserstein-inspired feedback. It shows how a low-rank Markov transition representation can be refined to better reflect true structure and interpretability, and how such refinements converge to meaningful solutions with spectral gap enhancement. This connects directly to issues in multi-head attention: -Both involve representing pairwise relationships (attention scores vs. transition affinities) in a full-rank or approximately low-rank form. -Low-rank factorization can produce spurious grouping or similarity structure that doesn’t reflect the true geometry, unless guided by careful feedback or regularization. -The GW-feedback orthogonalization that we exemplify with shows some ideas how to improve, stabilize, and interpret the geometry of such low-rank relational structures, offering insight into how optimization can go astray in ill-conditioned, low-rank parameter spaces, but also how the implication of the approximation can be assessed. Give it a read: Low-rank constraints are not just computational conveniences — they have real geometric and optimization implications, and without appropriate mechanisms (feedback, orthogonalization, spectral refinement), learned low-rank representations (like attention maps or transition matrices) can be misleading or unstable. I’d like to hear your thoughts on this. Is it needed? Should we care? What’s a good future architecture? What should I read? Merry Christmas!

10

I recently released on arXiv a new version of the paper The Problem of Social Cost in Multi-Agent General Reinforcement Learning: Survey and Synthesis, which can be found at https://lnkd.in/gbJ_dhMG The new version has - a description in Section 5.2 of a Bayesian reinforcement learning agent adapted from Dynamic Hedge AIXI that, as a group, will converge to a Nash equilibrium in general history-based environments as long as a certain “grain-of-truth” condition is satisfied; - a description in Section 5.3 on how the latest advances in Swap Regret Minimisation algorithms can be used to design agents that collectively converge to a correlated equilibrium when the “grain-of-truth” condition is not necessarily satisfied; and - a new Guaranteed Utility Mechanism as an alternative to the dynamic VCG-based Mechanism. These are all non-trivial extensions of the paper that build on recent new results in different fields and they are worth a read for those interested in the general topic. #AGISafety #SocialCost

17

New paper on improving LLMs' native capabilities of retrieval/ranking, to be presented in NeurIPS25: https://lnkd.in/eSX8vKPm Can we give LLM a list of items in context and ask it to retrieve/rank relevant ones? How does LLM solve such a problem? Is attention doing something similar to embedding-based retrieval? How to make it efficient with long context? Can we leverage embedding learning techniques to improve quality?

301

5 Comments