How to Use Memory Innovation in AI Hardware

I'm deep in the AI memory rabbit hole this week. Forget simple KV stores or fancy vector DBs acting like they've solved recall. Today’s deep dive is into MemOS, an open-source library that treats memory like a proper operating system framework with interfaces, operations, and infrastructure. Think of it as upgrading your agent's brain from sticky notes to a hypervisor managing cognitive resources. And yes, it's making my Qwen3 235B on-device runs significantly less... forgetful. Most projects out there often hyper-focus on external plaintext retrieval. MemOS integrates plaintext, activation, AND parameter memories – a proper memory hierarchy, not just a single-threaded fetch. It's like having RAM, cache, and disk, not just a single floppy drive. It doesn't just store memories; it manages them. Creation, activation (pulling into context), archiving (moving to cold storage), and expiration (the polite "forget this nonsense" signal). Full. Memory. Concierge. Service. Has fine-grained access control and versioning. Provenance tracking is baked into the data structure itself. No more wondering which hallucination spawned that terrible output or who gave the agent permission to recall your embarrassing internal docs. Audit trails are now a feature, not an afterthought. I’m watching MemOS automatically promote hot plaintext to faster activation memory (or demote cold activation back) based on usage patterns, and it’s pure sysadmin joy. It's like an LRU cache got a PhD in cognitive psychology and started optimizing itself. Efficiency? We got it. It works beautifully with serious on-device LLMs. I'm hammering it with Qwen3 235B locally, and the difference in coherent, context-aware persistence is noticeable. Less "wait, what were we talking about?", more "Ah yes, user, based on our conversation 47 interactions ago and the relevant archived parameter, I suggest..." Make sure you own your AI. AI in the cloud is not aligned with you; it’s aligned with the company that owns it.

Claude Code's source code leaked last week. 512,000 lines of TypeScript. Most people focused on the drama. I focused on the memory architecture. Here's how Claude Code actually remembers things across sessions — and why it's a masterclass in agent design: 𝗧𝗵𝗲 𝟯-𝗟𝗮𝘆𝗲𝗿 𝗠𝗲𝗺𝗼𝗿𝘆 𝗔𝗿𝗰𝗵𝗶𝘁𝗲𝗰𝘁𝘂𝗿𝗲: 𝗟𝗮𝘆𝗲𝗿 𝟭 — 𝗠𝗘𝗠𝗢𝗥𝗬. 𝗺𝗱 (𝗔𝗹𝘄𝗮𝘆𝘀 𝗟𝗼𝗮𝗱𝗲𝗱) A lightweight index file. Not storage — pointers. Each line is under 150 characters. First 200 lines get injected into context at every session start. It points to topic files. It never holds the actual knowledge. Think of it as a table of contents, not the book. 𝗟𝗮𝘆𝗲𝗿 𝟮 — 𝗧𝗼𝗽𝗶𝗰 𝗙𝗶𝗹𝗲𝘀 (𝗢𝗻-𝗗𝗲𝗺𝗮𝗻𝗱) Detailed knowledge spread across separate markdown files. Architecture decisions. Naming conventions. Test commands. Loaded only when MEMORY. md says they're relevant. Not everything gets loaded. Only what's needed right now. 𝗟𝗮𝘆𝗲𝗿 𝟯 — 𝗥𝗮𝘄 𝗧𝗿𝗮𝗻𝘀𝗰𝗿𝗶𝗽𝘁𝘀 (𝗚𝗿𝗲𝗽-𝗕𝗮𝘀𝗲𝗱 𝗦𝗲𝗮𝗿𝗰𝗵) Past session transcripts are never fully reloaded. They're searched using grep for specific identifiers. Fast. Deterministic. No embeddings. No vector DB. Just plain text search when the first two layers aren't enough. But here's the part that blew my mind: 𝗦𝗸𝗲𝗽𝘁𝗶𝗰𝗮𝗹 𝗠𝗲𝗺𝗼𝗿𝘆. The agent treats its own memory as a hint, not a fact. Memory says a function exists? → Verify against the codebase first. Memory says a file is at this path? → Check before using it. And one more design principle hidden in the code: If something can be re-derived from source code — it doesn't get stored. Code patterns, conventions, architecture? Excluded from memory saves entirely. Because if it can be looked up, it shouldn't be remembered. 𝗪𝗵𝘆 𝘁𝗵𝗶𝘀 𝗺𝗮𝘁𝘁𝗲𝗿𝘀 𝗯𝗲𝘆𝗼𝗻𝗱 𝗖𝗹𝗮𝘂𝗱𝗲 𝗖𝗼𝗱𝗲: This 3-layer pattern is model-agnostic. Any team building AI agents can steal this: → Keep your always-loaded context tiny → Reference everything else via pointers → Never persist what can be looked up → Treat memory as a hint, not truth The future of AI agents isn't about how much they remember. It's about how well they forget. What memory patterns are you using in your agent builds?

Launching Glance AI was never just an engineering challenge. It was a relentless tug-of-war between accuracy, user delight, and cost. Early on, we doubled down on delivering value and precision, knowing that at small scale, costs wouldn’t limit us. But as our user base exploded, and with every iteration running on GPUs, the team had to master every angle, from token optimization to splitting training from inference. The payoff? Glance AI now outperforms industry benchmarks for both cost-efficiency and fidelity. But our dependency on a single GPU class still lingered. So, when Jonathan Ross (the creator of Google’s pioneering TPU and now CEO of Groq) visited us, it gave us many more ideas. We’ve been experimenting with TPUs to streamline training and inference, but Groq’s chip, the Language Processing Unit (LPU), looks very promising. It’s a leap in AI hardware design, using colossal amounts of SRAM (up to 230MB per chip, nearly 10x more than top GPUs) and delivering unprecedented memory bandwidth (nearly 80TB/s, 25x higher than the best H100 GPUs). This means instantaneous data movement, blazing speeds, and a dramatic cut in bottlenecks. Here’s what blew me away about Groq: ➡ No DRAM bottleneck: All live data for inference stays in ultra-fast SRAM, eliminating DRAM/HBM delays and accelerating LLM responses. ➡ Single-core simplicity: Groq’s Tensor Streaming Processor ditches GPU multi-core complexity for streamlined, predictable workflows led by software instead of clunky hardware synchronizations. ➡ Assembly-line architecture: Compared to GPUs’ hub-and-spoke layout, data and compute flow seamlessly, making programming fast and dead times a thing of the past. ➡ Software-led execution: All planning is handled by software, liberating silicon for raw compute. Caching and hardware sync layers? Gone. More resources for solving problems, not shuffling data. ➡ Chip-level orchestration: Hundreds of Groq chips sync as one “virtual core," a feature that’s crucial for scaling huge LLMs. Beyond the pure speed and efficiency, Groq’s software-first approach is a paradigm shift, unlocking new possibilities for model deployment at scale. Now I am all for benchmarking Glance AI inference on Groq chips. Stay tuned: We will be sharing our results. Abhay Singhal | Arvind Jayaprakash | Debleena Das | Raman Srinivasan | Sudheer Bhat | Vivek Y S | Srikanth Sundarrajan InMobi InMobi Advertising Glance #AI #Hardware #Groq #Innovation #GlanceAI #EngineeringExcellence

We used to worry about mobile data limits - today, the tech world’s biggest anxiety is power. The skyrocketing energy consumption of GPUs during LLM inference isn't just an environmental concern, it's an engineering bottleneck. Standard infrastructure is incredibly wasteful. As someone deep in large-scale AI architecture, I knew we couldn't just keep throwing more GPUs at the problem. The real culprit isn't raw compute; it’s memory bandwidth and the KV Cache. When an LLM recalls conversation history, standard systems struggle with redundancy. They reload massive amounts of data or inefficiently access shared memory. Moving all that data between VRAM and the chip is exactly what drives up the wattage per token. We needed to rethink memory access entirely - that’s where my patent for Vectors and RadixAttention comes in. Instead of treating the KV cache as fragmented pages, RadixAttention uses a Radix Tree structure to index it. The game-changer? It recognizes shared context instantly. If multiple users query an LLM on the same document, that context is stored once and accessed by everyone, with zero redundant data movement. By fundamentally solving the KV cache redundancy problem, the impact was massive: ⚡️ Significantly Lower VRAM Usage: Eliminated duplicate storage, enabling larger models and more concurrent users on existing hardware. 🍃 Drastic Wattage Drop: Less data movement equals vastly less energy consumed per token. 🚀 Unprecedented Efficiency:Faster, radically more cost-effective inference at scale. The future of AI isn't just about building bigger models or faster chips, it's about designing smarter architecture. We can't ignore the energy bill of innovation. Proud to be building the infrastructure for a sustainable AI future.

LLMs have hit a memory wall — and our ASPLOS 2026 paper tackles it head-on. This work is a proud outcome of the IBM-RPI Future of Computing Research Collaboration (FCRC) co-supervised with Prof. @liu liu. Congratulations to lead author Zehao Fan and all co-authors: Yunzhen Liu, Garrett Gagnon, Zhenyu Liu, Yayue Hou, and Hadjer Benmeziane. Read the full paper: https://lnkd.in/dRRU7T7k For a technical deep dive on this work, check out my blog at: https://lnkd.in/dYKzsCMF During decoding, LLMs perform one matrix-vector multiply per token. The GPU spends more time waiting for data than computing. Processing-in-Memory (PIM) brings computation closer to data, but existing PIM designs assume dense attention and struggle with the irregular access patterns of sparse token retrieval. STARC solves this with a simple idea: cluster semantically similar key-value pairs and co-locate them in contiguous PIM memory rows. This makes sparsity hardware-visible — enabling real computation skipping at the memory level. Results: up to 93% latency reduction and 92% energy reduction on the attention layer, with no loss in model accuracy. #ASPLOS2026 #LLMInference #ProcessingInMemory #AIHardware #EfficientAI #IBMResearch #RPI #FCRC

I learned the most expensive lesson about AI memory when our production agent forgot a conversation it had 40 minutes earlier with the same customer. ‎ Memory is not a feature. It is the difference between a tool and a relationship. ‎ The customer had explained a complex billing dispute in detail. Our agent understood the whole situation. Proposed a resolution. The customer agreed. ‎ Then the session timed out. The customer called back. Got the same agent. And had to explain everything from scratch. ‎ She asked to speak to a human. When my team pulled the transcript, the agent's second interaction started with, "How can I help you today?" as if the previous 40 minutes had never happened. ‎ My head of AI in Bangalore looked at the data and said, "The agent has amnesia by design. We built it that way because memory was expensive." ‎ That sentence haunted me for weeks. We had optimized for cost and ended up destroying the customer experience. ‎ Here is what I learned about AI memory after deploying agents across three enterprise organizations. Memory is not one thing. It is a stack, and most teams only build the bottom layer. ‎ The context window is short term working memory. It gets you through a single conversation. The conversation buffer extends that to a session. But neither of those helps when a customer comes back two days later and expects you to remember who they are. ‎ The layer most teams skip is what I call institutional memory. The system's ability to learn from past interactions and apply those lessons to new ones. ‎ We rebuilt our agent architecture with four memory layers. Context window for the current turn. Session buffer for the conversation. Customer profile memory that persists across interactions. And a case library that stores successful resolutions so the agent can reference similar past situations. ‎ Cost increased 22%. Customer escalations dropped 48%. Average resolution time dropped by 11 minutes. ‎ The cheapest AI agent is not the one with the lowest token cost. It is the one that never makes the customer repeat themselves.

Building smarter AI agents that remember, learn, and evolve is very challenging, but not impossible. You can use this step-by-step framework to build AI agents with memory. Modern AI agents need more than just logic, they need context, personalization, and memory. This framework outlines the key stages to help you build memory-enabled agents that behave more like real assistants and less like static bots. 1. 🔸Start with Purpose: Define your agent’s role, whether an assistant, planner, or analyst; then map out the memory needs (short-term, long-term, episodic, semantic). 2. 🔸Select Your Tools: Pick your tech stack wisely whether it's LangChain, LlamaIndex, Zep, or OpenAI, each plays a distinct role in building memory pipelines. 3. 🔸Design the Memory System: From storage formats and access policies to retrieval methods and database selection (Pinecone, Weaviate), lay a strong foundation for efficient memory handling. 4. 🔸Add Intelligence with Retrieval & Graphs: Set up RAG pipelines and integrate knowledge graphs to ensure your agent reasons and responds with grounded context and structured facts. 5. 🔸Personalize with Profiles & Session Memory: Track user behavior across sessions. Store preferences, context, and feedback history for truly adaptive interactions. 6. 🔸Implement Loops & Feedback: Introduce memory loops for perception, reasoning, and reflection. Enable agents to learn from outcomes and adapt their responses accordingly. 7. 🔸Test, Secure, and Scale: Simulate edge cases, monitor memory quality, encrypt sensitive data, and continuously fine-tune the system as usage grows. 🔸🔸This guide gives you the complete blueprint for building reliable, scalable AI agents with memory as their competitive edge. 🔹🔹Save it and start building AI that doesn’t just respond, it remembers. #llm #genai #aiagents #artificialintelligence

The interview is for a Generative AI Engineer role at Cohere. Interviewer: "Your client complains that the LLM keeps losing track of earlier details in a long chat. What's happening?" You: "That's a classic context window problem. Every LLM has a fixed memory limit - say 8k, 32k, or 200k tokens. Once that's exceeded, earlier tokens get dropped or compressed, and the model literally forgets." Interviewer: "So you just buy a bigger model?" You: "You can, but that's like using a megaphone when you need a microphone. A larger context window costs more, runs slower, and doesn't always reason better." Interviewer: "Then how do you manage long-term memory?" You: 1. Summarization memory - periodically condense earlier chat segments into concise summaries. 2. Vector memory - store older context as embeddings; retrieve only the relevant pieces later. 3. Hybrid memory - combine summaries for continuity and retrieval for precision. Interviewer: "So you’re basically simulating memory?" You: "Yep. LLMs are stateless by design. You build memory on top of them - a retrieval layer that acts like long-term memory. Otherwise, your chatbot becomes a goldfish." Interviewer: "And how do you know if the memory strategy works?" You: "When the system recalls context correctly without bloating cost or latency. If a user says, 'Remind me what I told you last week,' and it answers from stored embeddings - that’s memory done right." Interviewer: "So context management isn’t a model issue - it’s an architecture issue?" You: "Exactly. Most think 'context length' equals intelligence. But true intelligence is recall with relevance - not recall with redundancy." #ai #genai #llms #rag #memory

Memory management is the trillion-dollar problem for enterprise AI. Why do you think OpenAI and Anthropic both launched memory features? Agents can pull useful context from Salesforce, Notion, Slack, and every other system, but that’s only the starting point. Most AI agents today are stateless. They retrieve data but never learn from it. Every query is day one again. To build high-performing AI systems in the enterprise, agents need to turn context into memory. They need to learn from their work, refine what matters, and adapt over time instead of treating every document as the source of truth. That requires a memory layer that is: - Persistent across sessions - Structured and queryable like a database - Selective, storing what matters and forgetting what doesn’t High-performing teams will control their memory layer, decide when and how context is passed in, and hold context back when it adds noise. Stateless agents with infinite context windows still fail. Stateful agents with managed memory improve with every interaction. At Hyperspell (YC F25), we’re building the memory layer that lets AI agents learn from experience the way people do.