Claudia Zhu

We just published our first technical research note. "Low-Latency Event-Driven Architecture for Real-Time Options Execution on IBKR" It covers how we eliminated compounding feed lag at SPY market open — from a growing 45-second backlog to stable sub-second bar sealing — through: → Queue-first callback offloading (Python GIL problem) → Per-venue DOM isolation across ARCA, ISLAND, IEX → Event-time binning across 6 independent intervals → Single source of truth for all technical indicators No signals. No alpha. Pure infrastructure. This is the kind of work that happens before a single trade fires — and the kind that determines whether your system survives the opening bell. 📄 Download: https://lnkd.in/g-XJsYwG #AlgoTrading #OptionsTrading #MarketMicrostructure #SystematicTrading #Python #QuantitativeFinance #TradingInfrastructure

1

Get faster and smarter MOE inference straight out of the box. 👇Deep dive on scaling expert parallelism with TensorRT-LLM. LLMs with MOE promise higher model capacity without linearly increasing compute costs - but they introduce new challenges -- more conditional computation, dynamic routing, and non-uniform GPU utilization -- solved by TensorRT-LLM. ✨ New in TensorRT-LLM has native support for expert parallelism—designed for fast, efficient inference with MoE models like Mixtral (Mistral AI) and DeepSeek (DeepSeek AI). This gives you: ✅ Dynamic expert routing: Automatically route tokens to the top-k experts with minimal overhead. ✅ Efficient expert scheduling: Balance expert loads across GPUs using smart sharding and token bucketization. ✅ Memory-aware execution: Maximize hardware utilization while respecting memory budgets. ✅ Drop-in support: Use @HuggingFace models with minimal code changes via TensorRT-LLM's #Python API. 🧠 How it works: MoE models activate only a subset of "experts" for each token. This dynamic nature is powerful—but hard to optimize. It’s all done under the hood using custom #CUDA kernels and NCCL-based communication primitives—giving you low latency, high throughput, and better GPU scaling. ✨ TensorRT-LLM handles: ✅Token-expert mapping using the gating network. ✅Token sorting to batch same-expert tokens together. ✅Expert parallel execution across GPUs. ✅Merging outputs for final predictions. 🛠️ Developer workflow - here is the code to get started. # Clone the repo git clone https://lnkd.in/g-GiDX23 # Use included examples to load and run a Mixtral model cd TensorRT-LLM/examples/mixtral From there, the Python API lets you load the model, convert it with TensorRT, and run expert parallel inference—all with a few lines of code. Results? 📈 Performance at scale. Tests show up to 2.3x faster inference throughput compared to standard tensor parallelism when using 8 GPUs and top-2 experts per token. Even better—TensorRT-LLM keeps efficiency high across increasing batch sizes. Want to see it in action or contribute? 👉 Read the full tech blog: https://lnkd.in/g_7YV3vV 👉 Explore the code on GitHub: https://lnkd.in/gNjQ5W2U 👉 Follow updates in the TensorRT-LLM repo: https://lnkd.in/gqSHYQ4u Share your experiences with us.

78

3 Comments

We (w/ Shansan Gong, Ruixiang ZHANG, Huangjie Zheng, Jiatao Gu, Navdeep Jaitly, Lingpeng Kong) released a family of 7B diffusion language models, DiffuCoder, that specializes on code generation, with a focus on understanding and improving masked diffusion models. A core analysis of DiffuCoder is the autoregressiveness (AR-ness) score, a novel metric that quantifies the causal patterns in decoding, revealing how diffusion models break from strict left-to-right generation for more flexible, non-linear code planning. Recent advances in autoregressive (AR) models dominate code generation, but diffusion-based LLMs (dLLMs) like DiffuCoder offer a promising alternative, especially for complex programming tasks. DiffuCoder explores how these models decode differently—showing less global AR-ness in code tasks compared to math—and how temperature affects both token selection and generation order, unlike traditional AR models. We also introduce coupled-GRPO, a post-training RL method with a coupled-sampling scheme, to reduce performance drops during accelerated decoding, boosting parallelism and efficiency. We use a self-improvement pipeline that leverages AR-ness analysis, coupled-GRPO optimization, and evaluation on benchmarks like AceCode-89k to refine decoding strategies. This approach enables DiffuCoder to navigate diverse code generation pathways and enhance performance with modest computational overhead. Looking ahead, we aim to further leverage Reinforcement Learning to steer code generation through these decoding patterns, with the discrete nature of AR-ness scores providing a foundation for search-based strategies—ideal for the sparse rewards of optimizing complex code structures. Check out our full paper and code for a deeper dive! Paper: https://lnkd.in/gVWU3BDJ Code: https://lnkd.in/gmXTZ_6n Models: https://lnkd.in/gTcKCDr9 #MachineLearning #AI #CodeGeneration #DiffusionModels #NLP

220

5 Comments

Most existing #lean4 datasets contain only correct proofs. Models learn error correction with RL, that's expensive. With UW Math AI lab we release a dataset of 260k erroneous Lean proofs with - compiler feedback - reasoning trace - corrected proof Improvements in Error Correction: - Goedel 8B: 2x - Kimina 8B: 3x Paper: https://lnkd.in/gaybt4bd

117

2 Comments

We at Haladir (YC W26) released a report on RLFR: Reinforcement Learning from Formally-Defined Rewards for Code Generation (https://lnkd.in/guCH7w7C). The core idea is simple: we present a simple offline and single-turn RL framework that uses formal verification as the primary reward signal for program synthesis in LLMs rather than unit test-based rewards (RLVR), showing improvements over a swath of different coding benchmarks. We fine-tune both Qwen2.5-Coder-7B-Instruct and Qwen3-8B using RLFR and Dafny, a specification language that utilizes Boogie (IVL) and Z3 (SMT solver) to check verification conditions. Despite solely training on Dafny, we show improvements on Python-native coding benchmarks, suggesting generalization past the formal verification domain.

90

8 Comments

Just shipped a full ML pipeline on Powerball data — from raw CSV to iOS app. 🚀 📓 Notebook now live on Kaggle: https://lnkd.in/eG-5ANw3 What I built: → 3 independent models: frequency analysis, recency-weighted stats, and a sliding-window MLP neural network → Weighted ensemble combining all signals into a single ranked pick → Chi-square goodness-of-fit test to measure how much the data deviates from uniform → Rolling stability simulation across all 1,064 draws → Full Docker setup for reproducibility → SwiftUI iOS app consuming the model output via CoreML Key finding: the lottery is exactly as random as it's supposed to be. The MLP accuracy confirms it. The p-values confirm it. Building the system to prove that is the whole point. Stack: Python · scikit-learn · pandas · matplotlib · Docker · Swift/SwiftUI · CoreML Would love feedback from anyone who's worked on similar time-series or sequence modeling projects. #MachineLearning #DataScience #Python #iOS #Kaggle #NeuralNetworks #Docker

9

👉 My Key Takeaways from Chip Huyen's Recent Interview on Lenny's Podcast Chip Huyen is the author of the widely recognized "AI Engineering: Building Applications with Foundation Models". Link to the podcast: https://lnkd.in/dxZ-tFWX 💡 Importance of post-training.  Pre-training gives you raw capabilities (next token prediction on massive data), but post-training is what makes the model actually usable. SFT on high-quality examples + RLHF. Fine-tuning should be your last resort, not first. Most problems can be solved with better prompts, better data, or RAG. 💡 Evals.  You can't improve what you can't measure. Need multiple types: unit tests (does this specific prompt work?), integration tests (does the whole pipeline work?), regression tests (did we break something?), and user feedback loops. The hardest part isn't writing evals; it's maintaining them as your product evolves. 💡AI products.  Reliability and UX matter more than models. Most AI product failures aren't about bad models: they're about reliability (API limits, latency spikes, poor monitoring) and UX (users don't understand how to use it, doesn't fit workflow). Building reliable platforms and talking to users constantly beats chasing SOTA models. Most insights come from watching users, not from benchmarks. 💡How to improve AI-powered apps.  What people think improves apps: staying current on AI news, chasing newest agentic framework, obsessing over vector database choice, constantly evaluating model benchmarks, fine-tuning models. What actually improves apps: talking to users, building reliable platforms, preparing better data, optimizing end-to-end workflows, writing better prompts. Better prompt engineering beats switching models 90% of the time. A well-crafted system prompt, clear instructions, good examples (few-shot), and proper output formatting can transform a mediocre experience into a great one. 💡 Advice for builders.  Start with user problem, not with cool AI technique. Use the simplest solution that works (often that's a good prompt, not a fine-tuned model). Build evals early. Focus on end-to-end experience. Don't fine-tune unless you've exhausted everything else. Don't treat AI as deterministic (it's not, you need to handle variability). Don't ignore data quality (garbage in, garbage out).

1

1 Comment

📢 LLMs can reason. But can they act, adapt, and collaborate over time? Researchers from UIUC, with collaborators from Meta, Amazon, and Google DeepMind, just released a deep survey on Agentic Reasoning for Large Language Models — and it reframes how we should think about “reasoning” altogether. The core shift: This work argues that strong benchmarks ≠ real intelligence. Instead of one-shot reasoning, agentic reasoning treats LLMs as interactive agents that: - Plan over long horizons - Use tools and external systems - Learn from feedback and memory - Coordinate with other agents They organize agentic reasoning into three layers: - Foundational → planning, tool use, search - Self-evolving → reflection, memory, continual improvement - Collective → multi-agent collaboration and role coordination Why this matters right now: - If you’re building or working with agents, this explains why: - Prompt tricks alone don’t scale to real-world tasks - Memory and feedback loops are becoming first-class design primitives - Multi-agent systems are less about “more agents” and more about reasoning structure It also clearly separates: - In-context reasoning (orchestration at inference time) vs Post-training reasoning (RL and fine-tuning to internalize behaviors) That distinction alone changes how you design agent systems. Feels like a roadmap for where agentic AI is actually headed — not just what demos look cool on Twitter. #AgenticAI #LLMResearch #AIEngineering #Meta #Amazon #Google

37

Just arXived: MUSTAFAR – Promoting Unstructured Sparsity for KV Cache Pruning in LLM Inference How do we shrink the memory-hogging key-value cache in large language models—without fine-tuning or accuracy degredation? Meet MUSTAFAR, our new end-to-end system that: 🔍 Prunes at the finest granularity with per-token magnitude-based unstructured sparsity: up to 70% pruning on both Key & Value caches with negligible quality drop. Notably, we can prune the Value cache (something structured methods can’t do) to very high sparsity! 📦 Compresses the resulting sparse cache into a bitmap-based format, packing the cache down to 45% of its original size. ⚡️ Accelerates decode-time attention via a custom sparse attention kernel, boosting tokens/sec throughput by up to 2.23× over dense inference. 🛠️ Integrates with orthogonal methods like token eviction and quantization for even richer compression. 👉 Check out the preprint & code: Paper: https://lnkd.in/eXeA99EZ Code & Kernel: https://lnkd.in/eZ9fyQP9 Huge thanks to my co-authors Helya Hosseini, Ramyad Hadidi and advisor Bahar Asgari for her guidance and insights 🚀

85

2 Comments

Read Chaim Rand's practical walkthrough for optimizing PyTorch inference. Learn about batched inference, memory formats, IPEX, and graph compilation on Intel Corporation Xeon.

5