Minchen Wang

Inference scaling is quietly becoming one of the most important ideas in modern LLMs. It lets you trade extra compute at inference time for major gains in accuracy, all without touching model weights.

𝗛𝗲𝗿𝗲’𝘀 𝗮𝗻 𝘂𝗻𝗲𝘅𝗽𝗲𝗰𝘁𝗲𝗱 𝘁𝗿𝘂𝘁𝗵 𝗮𝗯𝗼𝘂𝘁 𝗔𝗜 𝗮𝗴𝗲𝗻𝘁𝘀: 𝘁𝗵𝗲 𝘪𝘯𝘵𝘦𝘭𝘭𝘪𝘨𝘦𝘯𝘤𝘦🤖 𝗶𝘀𝗻’𝘁 𝘁𝗵𝗲 𝗵𝗮𝗿𝗱 𝗽𝗮𝗿𝘁 — 𝘁𝗵𝗲 𝘪𝘯𝘵𝘦𝘳𝘧𝘢𝘤𝘦 🖥️ 𝗶𝘀. We’ve reached a point where spinning up an agent backend is almost trivial. Frameworks like LangGraph, CrewAI, and Mastra have made orchestration feel plug-and-play. But the moment you try turning that backend into a real product, a different kind of complexity appears. You now need to handle: ⚡ Live streaming of tokens and events 🔄 Perfect state alignment between the agent and the UI 🧩 Structured parsing of actions, tool calls, and reasoning traces ♻️ Session continuity when someone reloads or switches devices This is the part teams end up rewriting over and over — custom WebSockets, reconnect logic, thread management, UI bridges… weeks of infrastructure engineering before users even see the agent. That’s why the new useAgent() hook from CopilotKit is interesting. With a single line: const { agent } = useAgent({ agentId: "my-agent" }); …you instantly get streaming, state sync, Human-in-the-Loop, AG-UI events, automatic thread persistence, and reconnection. No custom plumbing required. It works with any backend that follows the AG-UI protocol, which essentially defines how agents talk to users — the missing layer between intelligence and experience. For anyone building interactive, conversational, or workflow-driven agents, this kind of abstraction changes how fast you can ship. https://lnkd.in/gYvsezuF

13

Get faster and smarter MOE inference straight out of the box. 👇Deep dive on scaling expert parallelism with TensorRT-LLM. LLMs with MOE promise higher model capacity without linearly increasing compute costs - but they introduce new challenges -- more conditional computation, dynamic routing, and non-uniform GPU utilization -- solved by TensorRT-LLM. ✨ New in TensorRT-LLM has native support for expert parallelism—designed for fast, efficient inference with MoE models like Mixtral (Mistral AI) and DeepSeek (DeepSeek AI). This gives you: ✅ Dynamic expert routing: Automatically route tokens to the top-k experts with minimal overhead. ✅ Efficient expert scheduling: Balance expert loads across GPUs using smart sharding and token bucketization. ✅ Memory-aware execution: Maximize hardware utilization while respecting memory budgets. ✅ Drop-in support: Use @HuggingFace models with minimal code changes via TensorRT-LLM's #Python API. 🧠 How it works: MoE models activate only a subset of "experts" for each token. This dynamic nature is powerful—but hard to optimize. It’s all done under the hood using custom #CUDA kernels and NCCL-based communication primitives—giving you low latency, high throughput, and better GPU scaling. ✨ TensorRT-LLM handles: ✅Token-expert mapping using the gating network. ✅Token sorting to batch same-expert tokens together. ✅Expert parallel execution across GPUs. ✅Merging outputs for final predictions. 🛠️ Developer workflow - here is the code to get started. # Clone the repo git clone https://lnkd.in/g-GiDX23 # Use included examples to load and run a Mixtral model cd TensorRT-LLM/examples/mixtral From there, the Python API lets you load the model, convert it with TensorRT, and run expert parallel inference—all with a few lines of code. Results? 📈 Performance at scale. Tests show up to 2.3x faster inference throughput compared to standard tensor parallelism when using 8 GPUs and top-2 experts per token. Even better—TensorRT-LLM keeps efficiency high across increasing batch sizes. Want to see it in action or contribute? 👉 Read the full tech blog: https://lnkd.in/g_7YV3vV 👉 Explore the code on GitHub: https://lnkd.in/gNjQ5W2U 👉 Follow updates in the TensorRT-LLM repo: https://lnkd.in/gqSHYQ4u Share your experiences with us.

78

3 Comments

🚀 Excited to share two of our recent papers on E-commerce Search & Retrieval: 📄 Beyond Text: Aligning Vision and Language for Multimodal E-Commerce Retrieval arXiv: https://lnkd.in/gUWgEGRx 📄 Unified Learning-to-Rank for Multi-Channel Retrieval in Large-Scale E-Commerce Search arXiv: https://lnkd.in/gtP-kM5T Both works explore multimodal retrieval and unified ranking frameworks for improving large-scale product search.

45

1 Comment

Hello friends -- Here is a quick tutorial on how to make a custom export for your AI/BI dashboards in Databricks. This is particularly useful if you have a dashboard with html formatting. Thank you for reading!

17

Forecasting what matters: TimesFM-2.5 on the VN2 Inventory Challenge (with covariates) Short horizons. Intermittent demand. Real retail constraints. I just ran Google Research’s TimesFM 2.5 across all 599 SKUs of the VN2 Inventory Planning challenge—and then layered a small, transparent covariates step (store, product hierarchy, calendar, drift index) on top. What I learned: - Strong zero-/few-shot prior: TimesFM 2.5 is already a solid baseline without any training. - Covariates help where planners feel it most: a simple xreg layer reduced error and pulled bias closer to zero—crucial for inventory decisions. - Baselines still matter: a seasonal MA remains a great guardrail for calibration; routed SES/Croston/TBATS helps on very intermittent SKUs; LightGBM is competitive when you engineer horizon-safe features. - Keep it legible: the whole stack is understandable enough to discuss with planners (and to debug when reality shifts). Why post this? Because foundation models for time series are becoming practical. Kudos to Google Research (Yossi Matias) for pushing the field with TimesFM, and to Nicolas Vandeput for making datasets like VN2 (and earlier VN1) accessible for honest benchmarking. If you’re exploring retail/supply-chain forecasting: start with a strong prior, add the covariates you trust, and make bias everyone’s business. Happy to share notebooks and code. — Resources (in comments): repo + notebook, VN2 challenge, and a pointer to VN1. #TimeSeries #Forecasting #GoogleResearch #TimesFM #RetailAnalytics #InventoryPlanning #FoundationModels #DataScience #VN2 #VN1 #OpenSource

42

4 Comments

New short course: DSPy: Build and Optimize Agentic Apps DSPy is a powerful open-source framework for automatically tuning prompts for GenAI applications. In this course, you'll learn to use DSPy, together with MLflow. This is built in partnership with Databricks and taught by Chen Qian, co-lead of the DSPy framework. Many AI builders spend hours hand-tuning prompts. When given a set of evals, DSPy automates this process. It’s especially useful for optimizing prompts, including few-shot prompts, in complex agentic AI workflows. Further, if you switch an application to a newer LLM, performance can degrade if your prompts were optimized to the previous model. DSPy automatically optimizes the entire system for the new LLM as well, using just a few evaluation examples. This course teaches DSPy works, and best practices for using it. You’ll write programs using DSPy’s signature-based programming model, debug them with MLflow tracing -- to gain visibility into how different parts of a pipeline, as well as how the overall system, are performing -- and automatically improve their accuracy with DSPy Optimizer. Please sign up here: https://lnkd.in/gdjae8AX

4,672

190 Comments

In this post, we explore reinforcement fine-tuning (RFT) for Amazon Nova models, which can be a powerful customization technique that learns through evaluation rather than imitation. We'll cover how RFT works, when to use it versus supervised...

1