Jiaying Li

New short course: DSPy: Build and Optimize Agentic Apps DSPy is a powerful open-source framework for automatically tuning prompts for GenAI applications. In this course, you'll learn to use DSPy, together with MLflow. This is built in partnership with Databricks and taught by Chen Qian, co-lead of the DSPy framework. Many AI builders spend hours hand-tuning prompts. When given a set of evals, DSPy automates this process. It’s especially useful for optimizing prompts, including few-shot prompts, in complex agentic AI workflows. Further, if you switch an application to a newer LLM, performance can degrade if your prompts were optimized to the previous model. DSPy automatically optimizes the entire system for the new LLM as well, using just a few evaluation examples. This course teaches DSPy works, and best practices for using it. You’ll write programs using DSPy’s signature-based programming model, debug them with MLflow tracing -- to gain visibility into how different parts of a pipeline, as well as how the overall system, are performing -- and automatically improve their accuracy with DSPy Optimizer. Please sign up here: https://lnkd.in/gdjae8AX

4,672

190 Comments

Get faster and smarter MOE inference straight out of the box. 👇Deep dive on scaling expert parallelism with TensorRT-LLM. LLMs with MOE promise higher model capacity without linearly increasing compute costs - but they introduce new challenges -- more conditional computation, dynamic routing, and non-uniform GPU utilization -- solved by TensorRT-LLM. ✨ New in TensorRT-LLM has native support for expert parallelism—designed for fast, efficient inference with MoE models like Mixtral (Mistral AI) and DeepSeek (DeepSeek AI). This gives you: ✅ Dynamic expert routing: Automatically route tokens to the top-k experts with minimal overhead. ✅ Efficient expert scheduling: Balance expert loads across GPUs using smart sharding and token bucketization. ✅ Memory-aware execution: Maximize hardware utilization while respecting memory budgets. ✅ Drop-in support: Use @HuggingFace models with minimal code changes via TensorRT-LLM's #Python API. 🧠 How it works: MoE models activate only a subset of "experts" for each token. This dynamic nature is powerful—but hard to optimize. It’s all done under the hood using custom #CUDA kernels and NCCL-based communication primitives—giving you low latency, high throughput, and better GPU scaling. ✨ TensorRT-LLM handles: ✅Token-expert mapping using the gating network. ✅Token sorting to batch same-expert tokens together. ✅Expert parallel execution across GPUs. ✅Merging outputs for final predictions. 🛠️ Developer workflow - here is the code to get started. # Clone the repo git clone https://lnkd.in/g-GiDX23 # Use included examples to load and run a Mixtral model cd TensorRT-LLM/examples/mixtral From there, the Python API lets you load the model, convert it with TensorRT, and run expert parallel inference—all with a few lines of code. Results? 📈 Performance at scale. Tests show up to 2.3x faster inference throughput compared to standard tensor parallelism when using 8 GPUs and top-2 experts per token. Even better—TensorRT-LLM keeps efficiency high across increasing batch sizes. Want to see it in action or contribute? 👉 Read the full tech blog: https://lnkd.in/g_7YV3vV 👉 Explore the code on GitHub: https://lnkd.in/gNjQ5W2U 👉 Follow updates in the TensorRT-LLM repo: https://lnkd.in/gqSHYQ4u Share your experiences with us.

78

3 Comments

We developed an open-source AI tool to design concrete mixes that are stronger, more sustainable, and ready to build with faster—speeding up construction while reducing environmental impact. https://lnkd.in/gPCk8tCM But the impact of this AI tool is not just hypothetical! Amrize used Meta’s AI-based technologies to design a new low-carbon mix, and successfully deployed it in an at-scale slab-on-grade application at Meta's new data center in Rosemont, MN. Compared to the legacy mix, this new AI-designed mix is: 🦁 Stronger ⏱ Faster 🍃 Lower carbon ⏱ ️The ideal set time All this was achieved without needing any new materials, nor special equipment. Best of all, the AI is open-sourced. https://lnkd.in/g2KA7KZW This work was featured in a Meta engineering blog article published today! https://lnkd.in/gBU9HY8H

98

9 Comments

Breaking the Multi-Stage Paradigm: OneSearch Brings End-to-End Generative Retrieval to Industrial E-Commerce Researchers from Kuaishou Technology have deployed what they claim is the first industrial-scale end-to-end generative framework for e-commerce search, fundamentally rethinking how search systems handle hundreds of millions of product candidates. Traditional e-commerce search relies on multi-stage cascading architectures- recall filters billions of items down to thousands, pre-ranking narrows further, and ranking produces final results. OneSearch replaces this entire pipeline with a single generative model that directly outputs item candidates from user queries. The Technical Architecture: The system introduces three core innovations working in concert: - Keyword-Enhanced Hierarchical Quantization Encoding (KHQE): Instead of standard semantic IDs, the framework uses a hybrid approach. It extracts 18 structured attributes via Named Entity Recognition, then applies RQ-Kmeans for hierarchical clustering combined with Optimized Product Quantization for residual features. This dual-tokenization approach (4096-1024-512 for RQ-Kmeans, plus 256-256 for OPQ) achieves 68% independent coding rate while preserving both shared semantics and distinctive item attributes. - Multi-View Behavior Sequence Injection: User IDs are constructed from weighted behavior sequences rather than random hashing. Short-term sequences (recent clicks and queries) feed explicitly into the prompt, while long-term sequences (click, order, and search-relevant unit data) are compressed through Q-Former aggregation at each hierarchical level. This multi-perspective approach captures both immediate intent and stable preferences. - Preference-Aware Reward System: A three-stage supervised fine-tuning process handles semantic alignment, co-occurrence learning, and personalization sequentially. The adaptive reward system defines six behavioral hierarchy levels, from search-driven purchases down to random items, with calibrated CTR and CVR metrics. A hybrid ranking framework combines reward-guided training with direct user interaction feedback through list-wise DPO optimization. The Results: Deployed across Kuaishou's search scenarios (detail page, mall, homepage), the system achieves 1.67% item CTR improvement, 2.40% buyer increase, and 3.22% order volume gain. More striking: Model FLOPs Utilization jumps from 3.26% to 27.32%, while operational expenditure drops by 75.40%. The framework handles cold-start scenarios effectively- 3.31% CTR improvement for new items, 2.50% for new users What's particularly interesting is the architectural efficiency. By using encoder-decoder models with constrained beam search and eliminating stage-to-stage communication overhead, the system consolidates what traditionally required separate recall, pre-ranking, and ranking models into a unified generative process.

30

2 Comments

Took some time to refresh my knowledge on the latest large-scale recommendation modeling. Takeaways: • 📈 Scaling the classical deep-wide ranking models: Longer user sequences + richer feature interactions under tight latency. Bytedance’s Rankmixer and LONGER papers showcase masterful architecture design. • 🤖 Generative recommendations: Moving beyond next-token prediction to next-action prediction. 🔁 “Action Speaks Louder than Words” (Meta) — one model for both retrieval and ranking. 📄 “A Generative Re-ranking Model for List-level Multi-objective Optimization” (Taobao) — predicts engagement for an entire list, full-page optimization with some greedy tricks. 🎯 “Recommender Systems with Generative Retrieval” (Google) — a “true generative” approach using semantic IDs (powerful, though tricky for fresh content) to generate the content directly. The real bottleneck today isn’t model architecture anymore — it’s training and serving at scale. GPT-style pipelines (OpenAI, Gemini, Claude) are setting the benchmark, and RecSys engineers are rapidly adopting similar techniques. It won't surprising to see MMOE + post-training ideas to show up in generative recsys soon. Just as GPT lowered the barrier for NLP, transformers will lower the barrier for next-gen recommendation systems. As GPUs get cheaper and more powerful, definitely more companies will adopt them. 🚀

97

4 Comments

How a 7B Model(SLMs) Beat GPT-4o(LLMs) in Agentic Workflows 🤯📉 We often assume that bigger is better when it comes to LLMs, especially for complex reasoning tasks. In my latest video, I break down AgentFlow, a new framework where a 7B parameter model outperforms top-tier proprietary models (like GPT-4o) on complex, multi-turn agentic benchmarks. Here’s what I cover in the deep dive: 🔹 From Monolithic to Modular: Most tool-using LLMs try to do everything (reasoning + tool calling) in one monolithic process. AgentFlow decomposes this into four specialized modules: Planner, Executor, Verifier, and Generator, all coordinating through an evolving memory. 🔹 The "Secret Sauce": Flow-GRPO: The real innovation is in the training. Traditional offline Supervised Fine-Tuning (SFT) often leads to performance collapse in these systems. They introduce Flow-GRPO (Flow-based Group Refined Policy Optimization), a method that trains the "Planner" agent inside the multi-turn loop. It solves the sparse reward problem by broadcasting the final outcome reward back to every step of the reasoning chain. 🔹 The Results are Wild: By optimizing the planner "in-the-flow," their 7B model achieved massive gains over state-of-the-art baselines: +14.9% in Search tasks +14.0% in Agentic reasoning +14.5% in Math Crucially: It outperformed the ~200B parameter GPT-4o across all these domains. This is a must-watch if you are building AI agents or interested in how we can make smaller models significantly smarter through better system architecture and training. 📺 Watch the full breakdown here: https://lnkd.in/gSfPHbwh Github : https://lnkd.in/g4RaEcFg #AI #AgenticAI #LLM #MachineLearning #AgentFlow #OpenSource

25

Back in 2023, Kira Wang and I worked on using LLMs as a customer simulator. The idea was simple. Every year, Amazon Japan and our selling partners run countless promotions. The most canonical way in a vacuum to measure incrementality of these promotions are randomized experiments. However, in reality, there are always some other promotions running, always a slightly different variation you would like to test, and always the operational limitations to run an experiment. As it turns out, if you give them sufficient contexts (feature and outcome distribution, few-shot examples), right set of features, and even some finetuning on your seemingly unrelated historical data from previous promotions, LLMs can reasonably perform a simulation close to what predictive models can achieve. In our research, we benchmarked on a predictive task and a recommendation task based on randomized experiment data to show this. The work itself is quite dated (first draft in 2023) and limited to our internal, small datasets so I'm very curious how people can improve upon this approach both in terms of performance and scalability. Workshop: https://lnkd.in/gw2BprtD Paper: https://lnkd.in/gWEpwvGY Repo: https://lnkd.in/gZQqshVZ

50

Document-centric tasks sit at the core of many enterprise, business, and government workflows. Search is important, but the real challenge is going beyond retrieval, enabling systems to reason across documents, verify facts, and handle multi-step information tasks. Great to see a new model from Databricks moving in this direction. I expect we’ll see many more models, including open-weight ones, designed specifically for document-driven workflows, an area with huge potential across enterprise and government use cases.

23

1 Comment