Dong Meng

Rachit Jain's walkthrough shows how to profile kernels with PyTorch tools, import traces into NVIDIA Nsight or Intel Advisor, and read the slanted ceilings that separate compute-bound from memory-bound regions.

5

🧠 Breaking Through AI's Memory Paradigm Barrier Many of us have likely faced a fundamental architectural constraint that shaped our interaction and system design with LLMs: the rigid split between immutable parameter memory (the model's pre-trained knowledge) and ephemeral, stateless retrieval systems like RAG. This creates a challenging dichotomy - one end is crystallized knowledge that's expensive to update and prone to forgetting, while the other is a series of temporary patches lacking true integration or lifecycle management. A new paper, MemOS: A Memory OS for AI System, proposes a cognitive architecture that treats memory as a first-class, manageable resource, moving beyond the current limitations. What truly excites me is the concept of MemCubes, which are unified memory abstractions that bridge three distinct layers. 1. Parameter Memory: Deep semantic knowledge embedded in the model's weights. 2. Activation Memory: Dynamic in-the-moment inference states and KV-cache. 3. Plaintext Memory: Explicit knowledge fragments from external sources. Significantly, MemCubes aren't static. They can be combined, moved, and merged, allowing knowledge to shift easily between different types of memory. This structure reflects how human memory consolidates. Experiences gradually blend and strengthen over time. A memory can grow from a brief activation to a more stable parameter, similar to how our own memories become stronger through repetition or rehearsal. This has important implications. 1. Transferable Intelligence: Memory can be shared, making it possible to transfer "experience" between agents without expensive retraining. 2. Continuous Learning: It sets up systems that can progressively learn and adapt from each interaction. 3. Deep Personalization: Agents can be tailored by adjusting their memory to fit specific contexts and users. MemOS reportedly achieves state-of-the-art performance, outperforming strong baselines (mem0, LangMem, Zep, OpenAI-Memory), especially in complex multi-hop and temporal reasoning tasks. The authors term this new paradigm as "Mem-training" (beyond pre and post training) - where the continuous evolution of memory, not just sporadic parameter updates, drives capability advancement. It suggests we're nearing an inflection point where memory architecture could become more critical than raw model scale. This could represent a major shift from perception-based AI to true memory-driven intelligence. The future may not only focus on creating larger static models but also on developing systems that can genuinely learn from experiences. This foundation is essential for the truly adaptive, continuously learning agents of tomorrow. 🤗 📄 Paper: https://lnkd.in/gdHDrZEa #AI #MachineLearning #LLM #MemoryArchitecture #Research #Innovation

8

Compressing LLMs but worried about accuracy? 🎯 EoRA uses eigenspace low-rank approximation to compensate for errors—no retraining needed. A promising direction for scalable, task-adaptive LLMs. New paper from #NVIDIAResearch.

1

Our latest PyTorch community blog from the SGLang Team covers how SGLang advances hybrid Mamba plus Attention models with improved memory efficiency, prefix caching, speculative decoding, and serving performance. 📖 Read the blog: https://lnkd.in/gfDZkZWF #PyTorch #AIInfrastructure #SGLang #LLM

110

1 Comment

We are reimagining PyTorch on TPU (https://lnkd.in/gCrM-TZD). This approach still calls XLA but in a profoundly different way, enabling a solid eager mode. In the limit, this new approach will enable a true JIT approach with recompilation and re-optimization, online feedback-directed optimizations, avoidance of OOMs, support for HW fungibility, hyper-optimizations of kernels, and more - all with close to zero code changes!

341

2 Comments

Reinforcement Fine Tuning Reinforcement Fine-Tuning (RFT) is a technique for adapting LLMs to complex reasoning tasks like mathematics and coding. RFT leverages reinforcement learning (RL) to help models develop their strategies for completing a task, rather than relying on pre-existing examples as in traditional supervised fine-tuning. One RL algorithm, called Group Relative Policy Optimization (GRPO), is beneficial for tasks with verifiable outcomes and can work well even when you have fewer than 100 training examples. Using RFT to adapt small, open-source models can lead to competitive performance on reasoning tasks, giving you more options for your LLM-powered applications. This deeeplearning.ai provides an overview of GRPO with simple examples. Worth having a look 😀 #grpo #llm #reasoning #rl #rlhf #sft #predibase

2

In collaborative machine learning (CML), early contributors take on more risk and encourage participation from wait-and-see parties, but existing CML frameworks treat every party as if they join simultaneously. The #NeurIPS2025 paper of Jiangwei Chen, Nguyen Pham, Rachael Sim, Arun Verma, Zhaoxuan Wu, Chuan Sheng Foo, and Bryan Kian Hsiang Low proposes a time-aware CML framework that rewards parties not just for what they contribute, but when they contribute: •⁠ ⁠8 incentives extending fairness with time-aware notions that incentivize early participation while preventing low-quality early submissions. •⁠ ⁠2 reward distribution methods that theoretically satisfy all incentives and recover the #ShapleyValue in time-agnostic cases. •⁠ ⁠A practical recipe for constructing valuation functions and reward realization, with empirical validation on synthetic and real-world data. Find our more at our posters! #NeurIPS2025 Thu, Dec 4, Exhibit Hall C, D, E #1104 #EurIPS2025 Wed, Dec 3, Poster Stand #56 Paper: https://lnkd.in/g_yM6j39 #FederatedLearning

50