How to size GPUs for LLM inference with FlexAI Inference Sizer

"Your LLM inference is running out of GPU memory with long conversations. How do you fix it without losing performance?" Instant Thought : "Buy more GPUs" or "truncate context." ❌ The real answer: It's not model weights — it's the KV cache. 👉 KV cache grows linearly with tokens. 👉 A 7B model with 8K context = ~4GB KV cache alone. 👉 Idle users = idle GPU memory = wasted $$$. The secret to scaling isn’t bigger GPUs — it’s tiered cache offloading. GPU → CPU RAM → SSD → Distributed storage (based on access patterns) Reuse cache for 14x faster time-to-first-token (vs recomputing) Handle multi-user sessions without OOM errors 💡 “Keep everything in GPU until OOM.” 💡 “Tiered offloading with LMCache.” Scaling LLMs = 80% memory management, 20% compute. Offload smart. Serve more. #LLM #MachineLearning #Inference #GPU #PerformanceOptimization #AI #MLOps #KVCache #LLMScaling For details: https://lnkd.in/gVghhBYy

For more than half a century, computing has relied on the Von Neumann or Harvard model. Nearly every modern chip — CPUs, GPUs and even many specialized accelerators — derives from this design. Over time, new architectures like Very Long Instruction Word (VLIW), dataflow processors and GPUs were introduced to address specific performance bottlenecks, but none offered a comprehensive alternative to the paradigm itself. A new approach called Deterministic Execution challenges this status quo. Instead of dynamically guessing what instructions to run next, it schedules every operation with cycle-level precision, creating a predictable execution timeline. This enables a single processor to unify scalar, vector and matrix compute — handling both general-purpose and AI-intensive workloads without relying on separate accelerators. https://lnkd.in/ge3sBkMN #BeyondVonNeumann #TowardAUnified #DeterministicArchitecture

GPUs are ⚡️ hard to find. And when you find some, they are 💸 expensive. And then, most of them are sitting ⏳ idle anyway. We’ve all been there, me included: workloads competing for the same GPU, jobs waiting in line, efficiency going out the window. While the rest is idle... How about sharing the GPUs within your team and your apps? In her latest article, our Senior Software Engineer Katarzyna Kujawa breaks down two proven methods for maximizing GPU efficiency: - Multi-Instance GPU aka MIG <- my all-time fav - GPU time-slicing Worth a read if you’ve ever wrestled with GPU bottlenecks. It’s about making clusters leaner, faster, and less costly - something we’re all chasing. https://lnkd.in/e_b8c7fF

HighPoint launches the Rocket 7638D, the industry's first GPU-Direct NVMe architecture—enabling direct peer-to-peer data transfer between storage and GPU, bypassing CPU bottlenecks for lightning-fast AI/ML performance. #GPUDirect #NVMe #Storage #AI #ML #HPC #DataAcceleration #TechInnovation #ComputeStorageIntegration #powerelectronics #powermanagement #powersemiconductor https://lnkd.in/d8VMhWyC

Unlocking GPU & Multi-Core Performance on Edge Devices: Experimenting with 𝗢𝗽𝗲𝗻𝗔𝗖𝗖 Are you 𝗺𝗼𝗱𝗲𝗿𝗻𝗶𝘇𝗶𝗻𝗴 𝗹𝗲𝗴𝗮𝗰𝘆 𝗖/𝗖++ 𝗰𝗼𝗱𝗲 on 𝗲𝗱𝗴𝗲 𝗱𝗲𝘃𝗶𝗰𝗲𝘀 like the 𝗝𝗲𝘁𝘀𝗼𝗻 AGX Orin? It is not always necessary to go ahead with a brute-force CUDA rewrite! Even a 𝘁𝗿𝗶𝘃𝗶𝗮𝗹 𝗯𝗲𝗻𝗰𝗵𝗺𝗮𝗿𝗸𝗶𝗻𝗴/𝗽𝗿𝗼𝗳𝗶𝗹𝗶𝗻𝗴 is surprisingly inline with what we commonly see in production-grade codebases. Here are three common approaches that get profiled/benchmarked: 1. 𝗖𝗣𝗨 𝗢𝗻𝗹𝘆 (optimizations without ACC or CUDA) 2. Decorating code with 𝗢𝗽𝗲𝗻𝗔𝗖𝗖 based pragma directives 3. Rewrite as a 𝗖𝗨𝗗𝗔 program Obviously OpenACC ain’t 𝗻𝗼 𝘀𝗶𝗹𝘃𝗲𝗿 𝗯𝘂𝗹𝗹𝗲𝘁; however our results show that 𝗢𝗽𝗲𝗻𝗔𝗖𝗖 can provide a significant speedup of ~𝟱× compared to CPU-only serial computation, acting as the ideal 𝗺𝗶𝗱𝗱𝗹𝗲 𝗴𝗿𝗼𝘂𝗻𝗱 migration path. Of course a brute-force CUDA rewrite did come up with a ~10x speedup. OpenACC allows you to leverage GPU or multi-core ARM capabilities using simple pragmas, 𝗼𝗳𝗳𝗲𝗿𝗶𝗻𝗴 𝘀𝗰𝗮𝗹𝗮𝗯𝗶𝗹𝗶𝘁𝘆 𝘄𝗶𝘁𝗵𝗼𝘂𝘁 𝗰𝗼𝗺𝗽𝗹𝗲𝘅𝗶𝘁𝘆. OpenACC works on both 𝗱𝗶𝘀𝗰𝗿𝗲𝘁𝗲 𝗚𝗣𝗨𝘀 and 𝗖𝗣𝗨-𝗼𝗻𝗹𝘆 𝗔𝗥𝗠 𝘀𝘆𝘀𝘁𝗲𝗺𝘀 (e.g. Jetson AGX Orin), making it an invaluable tool to 𝘂𝗻𝗹𝗼𝗰𝗸 𝗺𝘂𝗹𝘁𝗶-𝗰𝗼𝗿𝗲 𝗽𝗲𝗿𝗳𝗼𝗿𝗺𝗮𝗻𝗰𝗲 with minimal code restructuring/refactoring (before you decide and go-ahead with a complete CUDA rewrite or the 𝘀𝘁𝗱::𝗲𝘅𝗲𝗰𝘂𝘁𝗶𝗼𝗻::𝗽𝗮𝗿 approaches (unsupported by nvc++ as on date)). Read the full document to see how OpenACC can accelerate Sensor Data Processing, Image Processing, and numerical code with 𝗦𝗶𝘅 𝗘𝗮𝘀𝘆 𝗣𝗶𝗲𝗰𝗲𝘀. https://lnkd.in/g4H4cyht #sixeasypieces #gpu #acc #openacc #pragma #cplusplus #c++ #nvcc #nvc++ #hpcsdk #nvidiahpcsdk #nvidia #onaquest #Quest1

“By predicting exactly when data will arrive — whether in 10 cycles or 200 — Deterministic Execution can slot dependent instructions into the right future cycle. This turns latency from a hazard into a schedulable event, keeping the execution units fully utilized and avoiding the massive thread and buffer overheads used by GPUs or custom VLIW chips.” #chipdesign #cpus #gpus

Experience extreme efficiency: Alibaba's new Qwen3-Next models bring 1M+ token context to commodity GPUs with FP8 quantization and MoE, beating Gemini-2.5-Flash. Alibaba's Qwen team has released FP8-quantized builds of their Qwen3-Next-80B-A3B models, enabling high-throughput, ultra-long context inference on commodity GPUs. The hybrid architecture, activating approximately 3B parameters per token, supports a native context of 262,144 tokens, scalable to over 1 million. FP8 offers deployment convenience and performance enhancements. * Alibaba's Qwen team released FP8 builds of Qwen3-Next-80B-A3B, enabling high-throughput, ultra-long context inference for MoE models on commodity GPUs. * The FP8 quantization significantly improves memory efficiency and inference throughput, especially for long sequences. * Deployment requires specific tools like sglang/vLLM nightly builds; users should validate FP8 accuracy and performance. The models reportedly outperform previous Qwen3 versions and even Gemini-2.5-Flash-Thinking on various benchmarks. Originally by Asif Razzaq — "Alibaba Qwen Team Just Released FP8 Builds of Qwen3-Next-80B-A3B (Instruct & Thinking), Bringing 80B/3B-Active Hybrid-MoE to Commodity GPUs" (Mon, 22 Sep 2025 10:04:21 +0000) — https://lnkd.in/eeCKAkUs What real-world applications could most benefit from Qwen3-Next-80B-A3B's ultra-long context and commodity GPU efficiency? #AlibabaQwen,#FP8Quantization,#MoEModels,#LLMInference

Microsoft has open-sourced bitnet.cpp, a blazing-fast 1-bit LLM inference framework optimized for CPUs — and it’s a big deal for local AI compute. This could redefine how we think about running large models without expensive GPUs or cloud dependencies. Key highlights: * Up to 6x faster inference with 82% lower energy consumption * 100B parameter models running directly on x86 CPUs (via kernel throughput demo) * Ternary weights (-1, 0, +1) + 8-bit activations for huge memory savings Alongside this, Microsoft also released BitNet b1.58 2B4T, the first open-source model using just 1.58 bits per weight — and it still performs impressively on benchmarks. If you care about efficient AI at scale, this is absolutely worth a look. The efficiency gains are real, though the “100B on CPU” demo was with dummy parameters (~5–7 t/s). The currently usable model is 2B4T — but the direction is clear. The era of efficient, low-bit AI might be closer than we think. GitHub: https://lnkd.in/gi6R8ptP Paper: https://lnkd.in/gzASgUaQ #AI #LLM #BitNet #OpenSource #EdgeAI #EfficientAI #Microsoft #MachineLearning #DeepLearning #AIResearch #GPU

Paying for powerful GPUs, but seeing disappointingly low utilization? 📉 It's a frustratingly common challenge. The problem is rarely the chip itself; it's a bottleneck hiding elsewhere in your system, silently killing your throughput. We asked Crusoe Senior Solutions Engineer, Martin Cala, for the first 3 things he checks when diagnosing underperforming AI infrastructure. Here’s his framework: 1️⃣ Is the data pipeline starving your GPU? If your storage and data loaders can't keep up, your GPU sits idle. This is the #1 cause of poor utilization and low throughput. 2️⃣ Are the nodes communicating efficiently? In distributed training, a slow network link forces GPUs to wait for data from their peers instead of computing. An InfiniBand or NVLink test can immediately pinpoint this issue. 3️⃣ Is the environment fully optimized? Simple misconfigurations or outdated drivers can leave massive performance on the table. A full system health check ensures your software isn't bottlenecking your hardware. Want to turn these diagnostics into a systematic, repeatable process? We compiled these expert checks and more into a comprehensive checklist. Download it now: https://lnkd.in/g-SUwQiw

LLM Inference at Scale: Why Bigger GPUs Won’t Save You When teams hit GPU OOM errors in long conversations, the default reaction is:
 “Buy more GPUs” or “truncate context.” Nope that’s wrong ! 
Both are short-term fixes , not real solutions. Here’s the truth:
The bottleneck isn’t the model weights , it’s the KV cache that grows linearly with every token. In production, this cache can easily consume 10x more memory than the model itself, while sitting idle between user interactions. That means: * High GPU utilization but low throughput = KV cache bottleneck * OOM errors = poor cache strategy, not insufficient hardware * Idle sessions = wasted GPU resources * No cache reuse = lost efficiency The scalable answer isn’t “bigger GPUs.”
It’s tiered KV cache offloading: * GPU → CPU RAM → SSD → Network storage (based on access patterns) * Multi-turn chats → CPU RAM * Document analysis → shared distributed cache * IDE/code sessions → local SSD * Batch inference → aggressive disk offload Production reality: * Great model + no cache plan = OOM crashes * Fancy hardware + poor cache mgmt = money burn * Smart offloading + cache reuse = scalable inference Tools like LMCache make this possible but the real edge comes from understanding memory hierarchy, access patterns, and cache hit rates. Scaling LLM inference isn’t about adding GPUs. It’s about engineering smarter memory strategies.