Monitoring LLM Performance Across Memory and GPUs

I used to think that quality of an LLM is determined by how well it has been trained. But later I realised, training happens once. Inference, happens millions of times every single day. What's the point of a well trained LLM if people are having a hard time using it. If you've observed, LLM sometimes "hangs" before it starts typing, or why the text appears at a specific speed. You’re looking at two fundamentally different mechanical battles happening inside the GPU. 1. The Prefill Phase (The Sprint) When you hit 'Enter,' the model processes your entire prompt in one go. The Goal: It builds a KV Cache (Key-Value cache) so it doesn't have to re-calculate your prompt for every new word. The Bottleneck: This is Compute-bound. It saturates the GPU with massive matrix multiplications. Metric to Watch: This determines your Time To First Token (TTFT). 2. The Decode Phase (The Marathon) Once the first word appears, the model switches gears to generate text sequentially, one token at a time. The Goal: Predict the next token using the growing context. The Bottleneck: It is Memory-bound. The GPU is actually waiting on the bandwidth to read the KV cache from memory. Metric to Watch: This determines your Inter-Token Latency (ITL). The Industry Shift: From "Bigger" to "Smarter" We are moving away from just throwing more GPUs at the problem. The industry is now obsessed with optimization strategies to break these bottlenecks: 1. Quantization: Shrinking model weights to fit into smaller memory footprints. 2. Speculative Decoding: Using a smaller "draft" model to guess tokens ahead of time, which the larger model then validates. 3. PagedAttention: Managing KV cache memory more like a computer’s RAM to reduce waste. The Bottom Line: Every AI response involves billions of parameters and millisecond-level optimization decisions. If you’re building AI products, your cost and user experience aren't just about the model size—they're about how you manage that memory-to-compute balance. Are you seeing the bottleneck in your projects? Are you optimizing for speed (TTFT) or throughput? Follow Bhavishya to stay upd-AI-ted with every scroll. #llm #agents #gpu

I am deploying my own LLM Mistral-7B-instruct with supercharged inference As I work on building a chat assistant with Mistral-7B to help customers navigate complex SAAS platform, I run into an important consideration, how will I scale and serve the LLM running the assistant. Let's look at a scenario: Using one GPU-A100 for deployment, our LLM Mistral-7B can generate 17 tokens per second. Now, lets say, if we have 1000 customers using our assistant at the same time, and average length of response from assistant is 150 tokens, putting the numbers together, our assistant will take 2 hours to process requests at anytime. An average reader's speed is 240 words per minute which we should match so our readers don't get bored but with the above setup, more than half the customers could even be waiting 1 hour to get any text at all. Not good at all for User Experience!! First, lets define the metrics we will use to assess the performance of LLM in the context of deployment: - Latency : Total time taken to process one user query. Important for better UX - Throughput: The number of tokens generated per second by the system. Important for scalability We are going to use a popular framework vLLM for optimization and benchmarking but lets look at the basic principles that vLLM leverages: 1. KV caching: - Transformer decoder architecture generates tokens sequentially and to generate a token, it uses all the past generated tokens. For each new token, a key-value vectors are generated which measures the relevance of the token to previous tokens. - So lets say, if we want to predict xth token, we will need KV vectors for 1...(x-1)th tokens, these vectors can be cached instead of regenerating them for every token, leading to time optimization with a memory trade-off. 2. Continuous batching our main optimization: - We parallelly process batches of customer queries, enhancing throughput. - However, differing response sizes in generative text lead to inefficient GPU memory use. - For examples: lets create a batch of two queries: - 'Delhi is the capital of which country?' -'Tell me about Harry potter' The first requires a brief response, while the second could be lengthy. With equal memory allocation per query, the GPU waits for the longer response to complete, leading to underutilized memory for the shorter query. This results in a hold-up of memory resources that could have been used for processing other queries. vLLM allows the efficient use of GPU memory to cache KV vectors, such that when a query in a batch is finished, another query can start processing in that batch. Observations on using vLLM on a batch of 60 queries: 1. Latency decreased more than 15x with vLLM 2. Throughput increased from 18 tokens/s to 385 tk/s 3. Throughput shows significant boost on large batches Link to reproduce results on colab: https://lnkd.in/ew_S_2WD If you are working on a similar project, you are welcome to share your experience :)

After training 𝐒𝐦𝐨𝐥𝐋𝐌𝟑 on 𝟑𝟖𝟒 𝐇𝟏𝟎𝟎𝐬 for nearly a month, I've come to realize something most people overlook: 𝐢𝐧𝐟𝐫𝐚𝐬𝐭𝐫𝐮𝐜𝐭𝐮𝐫𝐞 𝐢𝐬 𝐭𝐡𝐞 𝐦𝐚𝐤𝐞-𝐨𝐫-𝐛𝐫𝐞𝐚𝐤 𝐟𝐚𝐜𝐭𝐨𝐫 𝐢𝐧 𝐋𝐋𝐌 𝐭𝐫𝐚𝐢𝐧𝐢𝐧𝐠. 🔥 Everyone talks about model architecture and data quality. And yes, those matter immensely. But here's what nobody tells you: when your training run fails at 2 AM because of mysterious 𝐍𝐂𝐂𝐋 𝐞𝐫𝐫𝐨𝐫𝐬, or when your expensive GPU cluster is running at 𝟔𝟎% 𝐞𝐟𝐟𝐢𝐜𝐢𝐞𝐧𝐜𝐲, the problem isn't your model. It's most probably a 𝐦𝐢𝐬𝐮𝐬𝐞 𝐨𝐟 𝐭𝐡𝐞 𝐡𝐚𝐫𝐝𝐰𝐚𝐫𝐞. Questions that seemed simple but had no clear answers: Why is 𝐌𝐨𝐄 𝐭𝐫𝐚𝐢𝐧𝐢𝐧𝐠 𝐬𝐥𝐨𝐰𝐞𝐫 𝐭𝐡𝐚𝐧 𝐝𝐞𝐧𝐬𝐞 𝐦𝐨𝐝𝐞𝐥𝐬? Which 𝐍𝐂𝐂𝐋 𝐟𝐥𝐚𝐠𝐬 should we actually set? How often should we checkpoint without killing throughput? That's why we built 𝐓𝐡𝐞 𝐒𝐦𝐨𝐥 𝐓𝐫𝐚𝐢𝐧𝐢𝐧𝐠 𝐏𝐥𝐚𝐲𝐛𝐨𝐨𝐤 📖: a complete guide covering everything from model architecture and data curation to the SmolLM3 training marathon, post-training techniques, and crucially, the 𝐢𝐧𝐟𝐫𝐚𝐬𝐭𝐫𝐮𝐜𝐭𝐮𝐫𝐞 𝐥𝐚𝐲𝐞𝐫 that most teams get wrong. Here's what surprised us most: 𝘪𝘯𝘵𝘦𝘳𝘤𝘰𝘯𝘯𝘦𝘤𝘵 𝘵𝘰𝘱𝘰𝘭𝘰𝘨𝘺 𝘪𝘴 𝘢𝘭𝘮𝘰𝘴𝘵 𝘢𝘭𝘸𝘢𝘺𝘴 𝘮𝘪𝘴𝘶𝘯𝘥𝘦𝘳𝘴𝘵𝘰𝘰𝘥, and wrong configurations can silently destroy your GPU-to-GPU bandwidth. We spent weeks validating every layer of our AWS p5 system, and the results were eye-opening. 👀 We validated real vs theoretical bandwidth across the entire stack: 𝐇𝐁𝐌𝟑 𝐡𝐢𝐭𝐭𝐢𝐧𝐠 𝟑 𝐓𝐁/𝐬, 𝐍𝐕𝐋𝐢𝐧𝐤 𝟒.𝟎 𝐫𝐞𝐚𝐜𝐡𝐢𝐧𝐠 𝟕𝟖𝟔 𝐆𝐁/𝐬, 𝐏𝐂𝐈𝐞 𝐆𝐞𝐧𝟒 𝐚𝐭 𝟏𝟒.𝟐 𝐆𝐁/𝐬. Then we ran collective operations across 𝟏𝟐𝟖 𝐆𝐏𝐔𝐬 (16 nodes, 8xH100s each) and measured how performance degrades at scale: all-reduce drops from 𝟒𝟖𝟎 𝐆𝐁/𝐬 on a single node to 𝟑𝟐𝟎-𝟑𝟓𝟎 𝐆𝐁/𝐬 across 16 nodes. The good news? Once you understand what's happening, you can fix it. We documented everything: bandwidth measurements, annotated topology diagrams, troubleshooting workflows. And listed the tools you can use: 𝐧𝐯𝐛𝐚𝐧𝐝𝐰𝐢𝐝𝐭𝐡 for measuring communication paths, 𝐍𝐒𝐢𝐠𝐡𝐭 𝐂𝐨𝐦𝐩𝐮𝐭𝐞 for roofline analysis, step-by-step guides for debugging your specific setup. Infrastructure shouldn't be this invisible layer that only a handful of experts understand. When you can 𝐦𝐞𝐚𝐬𝐮𝐫𝐞, 𝐯𝐢𝐬𝐮𝐚𝐥𝐢𝐳𝐞, 𝐚𝐧𝐝 𝐝𝐞𝐛𝐮𝐠 𝐢𝐭 𝐩𝐫𝐨𝐩𝐞𝐫𝐥𝐲, suddenly those mysterious slowdowns become solvable problems. 🚀 If you've ever wondered why your training runs are slower than they should be, or you're planning to scale up and want to avoid expensive mistakes, this guide might save you weeks of debugging. 𝐓𝐡𝐞 𝐒𝐦𝐨𝐥 𝐓𝐫𝐚𝐢𝐧𝐢𝐧𝐠 𝐏𝐥𝐚𝐲𝐛𝐨𝐨𝐤: https://lnkd.in/e5MKXUHS Shared with ❤️ by the HuggingFace team

𝐒𝐜𝐚𝐥𝐢𝐧𝐠 𝐢𝐧𝐟𝐞𝐫𝐞𝐧𝐜𝐞 𝐦𝐞𝐚𝐧𝐬 𝐭𝐫𝐞𝐚𝐭𝐢𝐧𝐠 𝐊𝐕 𝐜𝐚𝐜𝐡𝐞 𝐚𝐬 𝐝𝐚𝐭𝐚, 𝐧𝐨𝐭 𝐣𝐮𝐬𝐭 𝐚 𝐛𝐮𝐟𝐟𝐞𝐫 The more we lean into multi turn LLM interactions, the more obvious it becomes that context does not come for free. I'm seeing more and more discussions on storage architecture for AI workloads, and it's about more than just IOPS. Every follow up question, every longer document, every agent step quietly grows the KV cache until it runs head first into GPU memory limits. At that point, you are not just tuning prompts, you are making infrastructure decisions. Great article from Ugur Kaynar, PhD and Gaurav Chawla demonstrating how we treat KV cache as a first class data problem, not just a side effect of the model. If that KV cache is forced to live only in limited GPU memory, you end up buying more GPUs just to hold context, not to actually do more work. 📈 𝐂𝐨𝐧𝐭𝐞𝐱𝐭 𝐠𝐫𝐨𝐰𝐭𝐡 𝐢𝐬 𝐚 𝐦𝐞𝐦𝐨𝐫𝐲 𝐚𝐧𝐝 𝐚𝐫𝐜𝐡𝐢𝐭𝐞𝐜𝐭𝐮𝐫𝐞 𝐩𝐫𝐨𝐛𝐥𝐞𝐦 As chat history and retrieved documents accumulate, the KV cache grows linearly with sequence length and batch size. The aggregate footprint across users quickly exceeds GPU memory, especially for large models and long running sessions. 🧠 𝐒𝐭𝐨𝐫𝐚𝐠𝐞 𝐚𝐰𝐚𝐫𝐞 𝐢𝐧𝐟𝐞𝐫𝐞𝐧𝐜𝐞 𝐜𝐡𝐚𝐧𝐠𝐞𝐬 𝐭𝐡𝐞 𝐩𝐥𝐚𝐲𝐛𝐨𝐨𝐤 Using vLLM, LMCache, and RDMA enabled Dell AI Storage Engines such as PowerScale, ObjectScale, and soon Project Lightning, the team has shown that you can push KV cache to shared storage, preserve multi turn context, and still reach significantly higher token throughput than GPU memory only setups, even as chat history expands. 🚦 𝐒𝐋𝐀 𝐚𝐧𝐝 𝐬𝐜𝐚𝐥𝐢𝐧𝐠 𝐜𝐨𝐧𝐜𝐮𝐫𝐫𝐞𝐧𝐜𝐲 𝐰𝐢𝐭𝐡𝐨𝐮𝐭 𝐣𝐮𝐬𝐭 𝐚𝐝𝐝𝐢𝐧𝐠 𝐆𝐏𝐔𝐬 By treating KV cache as something you can place, tier, and serve like any other critical dataset, the architecture keeps time to first token and throughput within target ranges, even with many users, long histories, and higher queries per second, instead of simply throwing more HBM at the problem. If you are thinking about how to make LLMs feel responsive at scale and how to build storage aware architectures, this KV cache centric view of inference is worth a closer look: https://lnkd.in/g4BjRYsW

We hit a wall with our A100 last month. 40GB VRAM, could barely serve 6 concurrent users without memory errors. Spent two days convinced it was a batching problem. Turns out it was a Memory fragmentation issue. Our KV cache allocator was treating GPU memory like it needed to be contiguous , leaving these massive unusable gaps between requests. Switched to vLLM which uses PagedAttention, basically the same virtual memory tricks from old school operating systems. Break memory into fixed pages, scatter them wherever there's space, use a block table to track everything. Now we're running 100+ concurrent users on the same GPU. Cost per million tokens dropped from ~$8 to under $0.50. Not flashy but this is the stuff that actually matters when you're trying to serve real traffic. Most LLM optimization content focuses on model architecture. Memory management is where some of the actual bottlenecks are. Wrote a deep dive on paged attentions here.  https://lnkd.in/gDFB3j34 #LLM #AIInfrastructure #MachineLearning #GPUOptimization #vLLM #AIEngineering #ProductionAI

𝗧𝗵𝗲 𝟮𝟬 𝘁𝗲𝗰𝗵𝗻𝗶𝗾𝘂𝗲𝘀 𝘀𝗲𝗽𝗮𝗿𝗮𝘁𝗶𝗻𝗴 𝗱𝗲𝗺𝗼-𝗴𝗿𝗮𝗱𝗲 𝗳𝗿𝗼𝗺 𝗽𝗿𝗼𝗱𝘂𝗰𝘁𝗶𝗼𝗻-𝗴𝗿𝗮𝗱𝗲 𝗟𝗟𝗠 𝗶𝗻𝗳𝗲𝗿𝗲𝗻𝗰𝗲. Most teams treat KV cache as a black box. That's a mistake. 𝘈𝘵 128𝘬+ 𝘤𝘰𝘯𝘵𝘦𝘹𝘵 𝘸𝘪𝘯𝘥𝘰𝘸𝘴, 𝗞𝗩 𝗰𝗮𝗰𝗵𝗲 is the single largest memory consumer in your inference stack — and the #1 bottleneck between you and production-grade LLM serving. I published an 𝙤𝙥𝙚𝙣 𝙞𝙢𝙥𝙡𝙚𝙢𝙚𝙣𝙩𝙖𝙩𝙞𝙤𝙣 𝙧𝙚𝙛𝙚𝙧𝙚𝙣𝙘𝙚 𝙘𝙤𝙫𝙚𝙧𝙞𝙣𝙜 20 𝙛𝙞𝙣𝙚-𝙜𝙧𝙖𝙞𝙣𝙚𝙙 𝙩𝙚𝙘𝙝𝙣𝙞𝙦𝙪𝙚𝙨 𝙖𝙘𝙧𝙤𝙨𝙨 6 𝙙𝙤𝙢𝙖𝙞𝙣𝙨: 𝗠𝗲𝗺𝗼𝗿𝘆 𝗔𝗿𝗰𝗵𝗶𝘁𝗲𝗰𝘁𝘂𝗿𝗲 → Multi-tier hierarchy design (HBM → DRAM → CXL → NVMe) → CXL.mem validation with numactl / cxl-cli → NUMA-aware memory binding (40-60% cross-node latency reduction) → DeepSpeed ZeRO-Infinity & vLLM offload configuration 𝗖𝗮𝗰𝗵𝗲 𝗠𝗮𝗻𝗮𝗴𝗲𝗺𝗲𝗻𝘁 → KV block swizzling (layer-major vs. token-major layouts) → Composable insertion/eviction hook interfaces → Attention-score weighted eviction with recency protection 𝗣𝗿𝗲𝗳𝗲𝘁𝗰𝗵 𝗦𝘁𝗿𝗮𝘁𝗲𝗴𝗶𝗲𝘀 → Sliding window prefetch for windowed attention models → Reuse distance-based prediction with EMA tracking → Three-stage non-blocking async pipeline (NVMe → pinned buffer → GPU) 𝗖𝗼𝗺𝗽𝗿𝗲𝘀𝘀𝗶𝗼𝗻 → LZ4 block-mode for <10ms latency budgets → ZSTD with dictionary training (3-5× ratio on cold tiers) 𝗛𝗮𝗿𝗱𝘄𝗮𝗿𝗲 𝗔𝗰𝗰𝗲𝗹𝗲𝗿𝗮𝘁𝗶𝗼𝗻 → fio + libaio benchmarking for SmartSSD I/O characterization → FPGA-offloaded compression (16 GB/s, zero CPU) → Programmable in-storage KV scoring for near-data processing 𝗩𝗮𝗹𝗶𝗱𝗮𝘁𝗶𝗼𝗻 → 128k context stress testing with tier utilization snapshots → Cold start, OOM graceful degradation, and session isolation tests → Synthetic trace generation (1M+) with replay comparison harness Every section includes production code, configuration guidance, decision heuristics, and performance benchmarks. This isn't theory. It's what the gap looks like between "we deployed a model" and "we operate an inference system." 🔗 𝗙𝘂𝗹𝗹 𝗶𝗻𝘁𝗲𝗿𝗮𝗰𝘁𝗶𝘃𝗲 𝗿𝗲𝗳𝗲𝗿𝗲𝗻𝗰𝗲: https://lnkd.in/gX2uUhjg #LLM #InferenceOptimization #KVCache #MLEngineering #SystemsDesign #AI #MachineLearning #GPU #DeepLearning

KV Cache is the silent killer eating your inference budget alive. If a 10-page document crashed your "production-ready" RAG system, this is likely the reason! I saw this exact failure pattern scaling recommendation systems at Twitter. Managing state is everything. In LLMs, that state is the KV Cache. The "Re-Reading" Problem: Without caching, LLMs re-compute attention for every previous token to generate the next word. It’s like re-reading a novel from page one every time you turn the page. The KV Cache stores vectors to solve this, but this "solution" creates a massive hidden tax. The Math is Unforgiving. For Llama-3 70B model with FP16 precision: → The Baseline: Model weights alone might take ~140GB (across GPUs). → The Cache: A single 4,000-token context consumes huge chunks of remaining VRAM. → The Reality: On an 80GB H100, after loading weights, you are fighting for scraps. The Real Bottleneck is *Fragmentation*. Traditionally, the cache requires contiguous memory blocks. Since you don't know if a response is 5 tokens or 500, you must over-allocate "just in case." This turns expensive GPU memory into "Swiss Cheese" - riddled with holes and reserved (but empty) slots. You end up with 50% of VRAM technically "free," but unusable because it's fragmented. We are facing a storage problem disguised as a math problem. The fix? Borrowing "Virtual Memory" logic from Operating Systems to "defrag" GPUs in real-time. It’s called PagedAttention. This is Part 2 of a deep dive into inference optimization. In the next one, I'll break down how PagedAttention can take GPU utilization from 40% to 92%.

Your LLM's biggest memory problem isn't the model itself. It's the conversation history. Every time a model reads your prompt, it keeps running notes on everything it's seen. At 128K tokens, these notes alone consume more GPU memory than the model. Google just published TurboQuant, a compression technique that shrinks these notes by 6x: -> Changes how the data is represented (polar coordinates instead of standard) -> Skips the expensive normalization step that limits traditional compression -> Works across models without any retraining or tuning The results: 6x memory reduction, 8x faster computation, zero accuracy loss across five benchmark suites. What this means practically: a session that burns 40GB of GPU memory drops to 7GB. Same model, same answers. 5-6x more concurrent users per card. Wrote the full breakdown as the first Sunday Deep Dive on my Substack. What changes for agent memory, RAG trade-offs, and inference pricing. #AIInfrastructure #LLM #Quantization

I learned these 4 concepts later than I should have. I’ve summarized everything you need to know in this post. For the longest part of my career in AI, I worked primarily with Deep Learning systems. I didn’t spend much time on tabular data, XGBoost, time-series forecasting, or recommender systems. Instead, I focused on AI for Video, Audio, and everything in between — from image generation to OCR to large-scale vision systems and robotics. But here's something I realized: Deployment strategies for deep learning models are very different from traditional ML. In fact, LLM systems feel a lot like deploying real-time vision systems. Both require high-performance inference, concurrency handling, and GPU optimization. The same principles apply: Optimized models. Optimized infrastructure. Optimized inference and GPUs. Distributed systems. But here’s where LLMs are different: inference is way more nuanced. In previous projects, I had to choose between real-time, batch, single-stage/ensemble, edge, or streaming inference. With LLMs, there are 4 distinct inference patterns you need to understand. Why? Because LLM inference happens in two phases: Prefill and Decode. ⇢ In the prefill phase, the model processes the entire input prompt and computes the key-value (KV) cache. ⇢ In the decode phase, it autoregressively generates tokens one at a time, updating the KV cache at each step. This matters because: 1/ Prefill is compute-bound (parallelizable and fast). 2/ Decode is memory-bound (sequential and slow, due to GPU memory traffic from KV cache updates). Understanding this gives you these four distinct LLM inference patterns: SISO – Short Input, Short Output → Low latency with fast generation. If using this pattern, aim to optimize for high concurrency. LILO – Long Input, Long Output → The most compute-intensive requires distributed inference, ideally with shared KV cache across nodes or tensor parallelism. LISO – Long Input, Short Output → This one has heavy prefill and quick decode. GPUs with enough VRAM would parallelize the prefill and generate fast. SILO – Short Input, Long Output → Light prefill, which runs fast but is heavier on decode. Decode latency dominates here. Takeaway: The longer the output, the more time is spent in the decode loop. Thus, before fine-tuning your own LLM and deploying it, keep in mind the inference pattern your system will target. Summary: - In LLM inference, there are 4 patterns. - For LISO, the generation and TFTT are fast. - On SISO, everything is fast. - For LILO, a lot of computing. - For SILO, prefill is fast, generation is slow. --- ♻️ Reshare and let others learn this too. Follow me to learn more about AI Engineering and production-ready AI Systems!