Chao Teng

We at Haladir (YC W26) released a report on RLFR: Reinforcement Learning from Formally-Defined Rewards for Code Generation (https://lnkd.in/guCH7w7C). The core idea is simple: we present a simple offline and single-turn RL framework that uses formal verification as the primary reward signal for program synthesis in LLMs rather than unit test-based rewards (RLVR), showing improvements over a swath of different coding benchmarks. We fine-tune both Qwen2.5-Coder-7B-Instruct and Qwen3-8B using RLFR and Dafny, a specification language that utilizes Boogie (IVL) and Z3 (SMT solver) to check verification conditions. Despite solely training on Dafny, we show improvements on Python-native coding benchmarks, suggesting generalization past the formal verification domain.

90

8 Comments

We (w/ Shansan Gong, Ruixiang ZHANG, Huangjie Zheng, Jiatao Gu, Navdeep Jaitly, Lingpeng Kong) released a family of 7B diffusion language models, DiffuCoder, that specializes on code generation, with a focus on understanding and improving masked diffusion models. A core analysis of DiffuCoder is the autoregressiveness (AR-ness) score, a novel metric that quantifies the causal patterns in decoding, revealing how diffusion models break from strict left-to-right generation for more flexible, non-linear code planning. Recent advances in autoregressive (AR) models dominate code generation, but diffusion-based LLMs (dLLMs) like DiffuCoder offer a promising alternative, especially for complex programming tasks. DiffuCoder explores how these models decode differently—showing less global AR-ness in code tasks compared to math—and how temperature affects both token selection and generation order, unlike traditional AR models. We also introduce coupled-GRPO, a post-training RL method with a coupled-sampling scheme, to reduce performance drops during accelerated decoding, boosting parallelism and efficiency. We use a self-improvement pipeline that leverages AR-ness analysis, coupled-GRPO optimization, and evaluation on benchmarks like AceCode-89k to refine decoding strategies. This approach enables DiffuCoder to navigate diverse code generation pathways and enhance performance with modest computational overhead. Looking ahead, we aim to further leverage Reinforcement Learning to steer code generation through these decoding patterns, with the discrete nature of AR-ness scores providing a foundation for search-based strategies—ideal for the sparse rewards of optimizing complex code structures. Check out our full paper and code for a deeper dive! Paper: https://lnkd.in/gVWU3BDJ Code: https://lnkd.in/gmXTZ_6n Models: https://lnkd.in/gTcKCDr9 #MachineLearning #AI #CodeGeneration #DiffusionModels #NLP

220

5 Comments

Just arXived: MUSTAFAR – Promoting Unstructured Sparsity for KV Cache Pruning in LLM Inference How do we shrink the memory-hogging key-value cache in large language models—without fine-tuning or accuracy degredation? Meet MUSTAFAR, our new end-to-end system that: 🔍 Prunes at the finest granularity with per-token magnitude-based unstructured sparsity: up to 70% pruning on both Key & Value caches with negligible quality drop. Notably, we can prune the Value cache (something structured methods can’t do) to very high sparsity! 📦 Compresses the resulting sparse cache into a bitmap-based format, packing the cache down to 45% of its original size. ⚡️ Accelerates decode-time attention via a custom sparse attention kernel, boosting tokens/sec throughput by up to 2.23× over dense inference. 🛠️ Integrates with orthogonal methods like token eviction and quantization for even richer compression. 👉 Check out the preprint & code: Paper: https://lnkd.in/eXeA99EZ Code & Kernel: https://lnkd.in/eZ9fyQP9 Huge thanks to my co-authors Helya Hosseini, Ramyad Hadidi and advisor Bahar Asgari for her guidance and insights 🚀

85

2 Comments

Most existing #lean4 datasets contain only correct proofs. Models learn error correction with RL, that's expensive. With UW Math AI lab we release a dataset of 260k erroneous Lean proofs with - compiler feedback - reasoning trace - corrected proof Improvements in Error Correction: - Goedel 8B: 2x - Kimina 8B: 3x Paper: https://lnkd.in/gaybt4bd

117

2 Comments

Today's AI/ML News 🤖💻 Researchers are advancing AI capabilities with new models and tools! Microsoft and Tsinghua University introduced Reward Reasoning Models, enhancing Large Language Models' ability to judge with reasoning, while the Synthetic Data Vault offers a step-by-step guide to creating synthetic data. LLM Reasoning Boosted Can LLMs Really Judge https://lnkd.in/gXu2rPbP 🤖 Researchers from Microsoft and Tsinghua University introduced Reward Reasoning Models, enhancing Large Language Models' ability to judge with reasoning. Synthetic Data Made Easy Step-by-Step Guide https://lnkd.in/gcarYFbS 🚀 The Synthetic Data Vault offers a step-by-step guide to generating realistic tabular data using machine learning. #AI #ML #LargeLanguageModels #SyntheticData #Innovation 🤖💻

6

𝗧𝗵𝗲 𝗯𝗲𝘀𝘁 𝗔𝗜 𝗰𝗼𝗱𝗶𝗻𝗴 𝗹𝗲𝘀𝘀𝗼𝗻 𝗰𝗼𝘀𝘁 𝗼𝘂𝗿 𝗲𝗻𝗴𝗶𝗻𝗲𝗲𝗿 $𝟲𝟬𝟬 𝗮𝗻𝗱 𝗮 𝗺𝗮𝗿𝗿𝗶𝗮𝗴𝗲 𝗮𝗿𝗴𝘂𝗺𝗲𝗻𝘁. Our VP of Engineering, Xiaofan(James) Luan, was supposed to buy his wife a Dior bag for their anniversary.  Instead, he bought three Claude Code subscriptions and spent the holiday trying to cross-compile 2 million lines of C++. Every fix on one platform broke two others.  $600 later, the only output was "git reset --hard" — and a very cold dinner table.😂 "Make it compile on Windows" is a trap. The real goal was "compile everywhere without hacks" — no AI is going to figure that out for you at 2 am. What worked: constraints before code, review tests not code, bottom-up, one layer at a time. Same task, two days. Then he ran six parallel Claude sessions across three machines with git worktree. The bottleneck stopped being intelligence and started being how fast one person can alt-tab. AI solves exactly the problem you give it. Engineering is in knowing which one to give. His wife is still waiting for that bag. Full story: https://lnkd.in/gtsW_Wvk ——— Follow Milvus, created by Zilliz, for everything related to unstructured data

9