Rajeev Bhatia

The paper “Challenges and Research Directions for LLM Inference Hardware” by Patterson and Ma frames what the industry is already talking about - inference is becoming constrained less by compute and more by memory bandwidth and capacity. As model sizes push toward 10 trillion parameters, an HBM-only strategy becomes economically and physically untenable. The authors argue for a tiered memory hierarchy and position High Bandwidth Flash (HBF) as a capacity-oriented extension of the memory system rather than as traditional storage. HBF borrows the 3D-stacking model from HBM but applies it to NAND flash. Instead of a discrete SSD controller connected to flash over a narrow interface, HBF integrates logic and NAND dies into a single vertical stack. A logic base die - similar in role to HBM’s base layer - implements the PHY, multi-channel flash controllers, and ECC. Vertical TSVs run through the stack, enabling thousands of parallel data paths and aggregate read bandwidths of ~1.6 TB/s per stack. A single HBF stack can provide ~512 GB of capacity, compared to 48–64 GB per HBM4 stack. The interface uses wide, synchronous, DDR-style signaling to the accelerator - similar to HBM. There are clear trade-offs, though. HBF operates at microsecond-scale latencies. Accesses are page-based (kilobytes), and small random reads quickly degrade effective bandwidth. Write endurance is limited (on the order of 10⁵ cycles), making HBF unsuitable as a general-purpose cache or a write-heavy tier. For this reason, HBM continues to remain critical for KV cache, activations, and other latency-sensitive accesses. Some systems are tiering KV cache - keeping hot KV in HBM while spilling colder data into CPU-attached LPDDR over coherent high-speed interconnects. HBF fits naturally with read-mostly data: model weights and slowly changing retrieval corpora. Weights are large, read-bandwidth-hungry, and not written during inference. Increasing per-node weight capacity by an order of magnitude reduces the number of accelerators required to fit a model, directly impacting system scale, power, and network complexity. HBF has shifted from theory to a formal industry roadmap following the 2025 SanDisk and SK Hynix alliance, with standardization underway and sampling slated for late 2026, targeting a full market debut on inference platforms by early 2027. Kioxia is pursuing an incremental path, moving multi-terabyte flash closer to accelerators over PCIe 6.0 with aggressive prefetching. Different implementations, same direction: flash is being pulled into the memory hierarchy! HBF is a serious attempt to address the capacity bottleneck. The open questions are no longer about whether this makes sense, but how to architect systems around it - software scheduling of page-based reads to hide microsecond latencies, balancing between HBM/HBF for different inference workloads, reducing tail latency, etc. Overall, I am excited to see more innovation for the memory wall problem!

156

6 Comments

Meet BeTaL: an LLM-in-the-loop framework that automates benchmark design. LLMs reason, simulate, and refine to create dynamic benchmarks that evolve as fast as the models they test—2–4× more precise than traditional methods. 📄 Research: https://lnkd.in/gRqAYtd9

62

1 Comment

In this episode, we discuss ImplicitQA: Going beyond frames towards Implicit Video Reasoning by Sirnam Swetha, Rohit Gupta, Parth Parag Kulkarni, David G Shatwell, Jeffrey A Chan Santiago, Nyle Siddiqui, Joseph Fioresi, Mubarak Shah. The paper introduces ImplicitQA, a new VideoQA benchmark designed to evaluate models on implicit reasoning in creative and cinematic videos, requiring understanding beyond explicit visual cues. It contains 1,000 carefully annotated question-answer pairs from over 320 narrative-driven video clips, emphasizing complex reasoning such as causality and social interactions. Evaluations show current VideoQA models struggle with these challenges, highlighting the need for improved implicit reasoning capabilities in the field.

14

3 Comments

Ruokai Yin, our summer intern from Yale to Microsoft Azure, worked with Xuan Zuo, @Sattwik Deb Mishra, Hokchhay T. and Preyas Shah and me. We published a paper based on his project: https://lnkd.in/eHuG3BSt Summary of paper: Finding optimal sharding is hard especially for large LLMs. A large number of devices have to be divided into TP, PP, EP dimensions. For TP, we have to decide the tensor dimension e.g. head, hidden size to shard for every op. Additionally, TP can be one-dimensional/two-dimensional etc. This makes for a large sharding space that is hard to cover exhaustively. SoTA methods use heuristics to select sharding e.g. GSPMD, Sequence Parallelism, Helix etc. There are significant time gaps, months or years, between the discovery of each of these methods. We want to fast-forward the process of discovery by 1) using a formal representation of the sharding space, and, 2) using reinforcement learning to search the sharding space.    This paper demonstrates that our method is 3.5x better than random search, and on par with hand-tuned heuristics when looking only at 1D sharding and without allowing sequence parallelism. We aim to extend the work to multi-dimensional sharding as well as sequence parallelism. Beyond that, we wish to combine the problem of sharding with fusion, tiling and collectives as well.

58

Just wrapped the 7th Perplexity AI Business Fellowship session featuring Ali Ghodsi, CEO of Databricks, in conversation with Aravind Srinivas. Ali's refreshing candor about AI's enterprise impact, dropping some uncomfortable truths about enterprise AI that every engineering leader needs to hear: - "The only reason the AI bubble hasn't burst is inference-time compute" - not actual breakthroughs - When Databricks plugged advanced reasoning models into enterprise applications? "Barely any boost." The problem isn't intelligence but context - is "C" California or Celsius? - Success formula: unique data + distribution channels. Nothing else matters. - AI revolutionizes coding, which represents just 20% of engineering time. The other 80%? Human meetings and alignment work which AI can't touch. - Biggest blocker? Not capabilities but evaluation frameworks. "We have no way to measure reliability" in specialized enterprise contexts. This mirrors what we're seeing in healthcare AI - where reliable assessment frameworks are critical when patient outcomes are on the line. As I noted in my analysis of BVP's Healthcare AI Index (https://lnkd.in/gy3P7zdU), success isn't about algorithms but workflow integration and evaluation rigor. For engineering teams, the path forward isn't chasing fancier models but solving the semantic understanding and evaluation challenges that remain stubbornly human. #EnterpriseAI #EngineeringLeadership #HealthcareAI

24

New research paper was recently published, which demonstrates a technique called type-constrained decoding in LLMs for codegen, reducing compilation errors by more than half. It’s an evolution of the constrained decoding technique used in ensuring JSON schema in LLM structured outputs. The algorithm: * The LLM produces a probability distribution over the next token * A token is sampled from this distribution * The completion engine checks if this token keeps the partial program in a state where it could still become well-typed * If not, that token’s probability is set to zero, the distribution is renormalized, and sampling occurs again * This continues until a valid token is found This will bring the already great coding models to the next level. Can't wait for Anthropic, OpenAI and Google implement it in their models in addition to structured outputs with JSON schema. #llm #ai #codegen

6

4 Comments

🚀 Act Three of my Hypermodern GenAI series is live! This chapter dives into LangChain’s Deep Agents and what they mean for the future of agent frameworks. From shallow vs deep agents to reasoning LLMs, I explore how these patterns shape the way we actually build and ship GenAI apps. The series isn’t about chasing hype — it’s about asking the hard questions: 🔹 When should we follow the defaults? 🔹 When does “depth” help, and when does it just add overhead? 🔹 How do we balance user experience, developer productivity, and system cost? Think of it as a guidebook for hypermodern generative AI agents: rethinking the playbook, experimenting carefully, and building with the user in mind.

3

1 Comment

Anthropic's recent blogpost discusses the important shift from evaluating simple LLMs ---> testing complex AI agents.  It highlights why standard benchmarks fall short for agentic workflows and offers a practical roadmap for building robust, multi-turn evaluations." Packed with insights, but couple of takeaways that stood out to me:  🎯 1. 𝐂𝐨𝐦𝐩𝐨𝐮𝐧𝐝𝐢𝐧𝐠 𝐯𝐚𝐥𝐮𝐞 𝐨𝐟 𝐭𝐡𝐞𝐬𝐞 𝐬𝐲𝐬𝐭𝐞𝐦𝐬: where costs are visible upfront but their benefits accumulate over time. (quicker iterations, faster deployment) 🎯 𝟐. 𝐓𝐡𝐞 𝐂𝐨𝐧𝐬𝐢𝐬𝐭𝐞𝐧𝐜𝐲 𝐅𝐢𝐱 / 𝐃𝐞𝐭𝐞𝐫𝐦𝐢𝐧𝐢𝐬𝐦 𝐢𝐧 𝐄𝐯𝐚𝐥𝐮𝐚𝐭𝐢𝐨𝐧𝐬: Regardless of agent type, agent behavior varies between runs. Sometimes, we want to measure how often (what proportion of the trials) an agent succeeds for a task, while for others, we want to measure if Agent succeeds atleast once. 🚀 Single success is enough across runs ? Use pass@k. 🚨 Need reliability across runs? Use pass^k. The post contains many more valuable insights worth exploring. Read more here: https://lnkd.in/gwCxqhDj

59

2 Comments

Compressing LLMs but worried about accuracy? 🎯 EoRA uses eigenspace low-rank approximation to compensate for errors—no retraining needed. A promising direction for scalable, task-adaptive LLMs. New paper from #NVIDIAResearch.

1

Comparing models.. Finding that a small 1B #llm hallucinates less than a 4B for semantic to structural matching. Small llms for targeted tasks for the win. The new Meta llama3.2 1B is better than expected and cheap on token cost.

3