Benefits of Caching Techniques

Caching Architecture Is the New Backbone of LLM Systems Performance, cost, and latency all depend on it. If your LLM bill is rising every month, you’re not alone. More usage More tokens More cost But here’s the catch. Most of that compute is repeated work. Same prompts Same context Same patterns And we recompute everything...every time... This is where inference caching changes the game. Not a new model Not a new architecture Just smarter reuse There are three layers that matter: 1. KV Caching - Happens inside the model - Stores attention states during generation - Prevents recomputing tokens within a request You’re already using it. You just don’t see it. 2. Prefix Caching - Extends this across requests - If your system prompt or reference context is constant, process it once → reuse it Simple rule Static content at the top Dynamic content at the end High impact. Almost zero effort 3. Semantic Caching - This is where things get interesting - Store past queries and responses - Retrieve based on meaning, not exact match In many cases, you can skip the LLM call entirely. Massive cost savings for support bots, FAQs, repeated queries. The real power comes from layering them - KV runs by default - Prefix reduces repeated context cost - Semantic avoids calls altogether Most teams focus on model quality. But in production, efficiency is what scales. Because in real systems: The cheapest token is the one you never generate.

𝗡𝗼𝘁 𝗲𝘃𝗲𝗿𝘆 𝗽𝗲𝗿𝗳𝗼𝗿𝗺𝗮𝗻𝗰𝗲 𝗽𝗿𝗼𝗯𝗹𝗲𝗺 𝗻𝗲𝗲𝗱𝘀 𝗮 𝗯𝗶𝗴𝗴𝗲𝗿 𝗲𝗻𝗴𝗶𝗻𝗲. 𝗦𝗼𝗺𝗲𝘁𝗶𝗺𝗲𝘀 𝗶𝘁 𝗷𝘂𝘀𝘁 𝗻𝗲𝗲𝗱𝘀 𝗮 𝘀𝗺𝗮𝗿𝘁𝗲𝗿 𝗽𝗹𝗮𝗰𝗲 𝘁𝗼 𝗽𝘂𝘁 𝘁𝗵𝗲 𝗮𝗻𝘀𝘄𝗲𝗿. Caching is the ultimate act of compute avoidance — placing pre-computed or frequently accessed data closer to where it's consumed, so the system doesn't repeat expensive work on every request. 𝗪𝗵𝗲𝗿𝗲 𝗰𝗮𝗰𝗵𝗶𝗻𝗴 𝗹𝗶𝘃𝗲𝘀 𝗶𝗻 𝗱𝗮𝘁𝗮 𝘀𝘆𝘀𝘁𝗲𝗺𝘀: → 𝗤𝘂𝗲𝗿𝘆 𝗰𝗮𝗰𝗵𝗲: Database stores results of recent queries. Same query, same parameters? Serve cached result instead of re-scanning. → 𝗠𝗮𝘁𝗲𝗿𝗶𝗮𝗹𝗶𝘇𝗲𝗱 𝘃𝗶𝗲𝘄𝘀: Pre-computed query results that refresh on a schedule. Caching at the SQL layer fast reads from a pre-built table instead of real-time JOINs. → 𝗔𝗽𝗽𝗹𝗶𝗰𝗮𝘁𝗶𝗼𝗻 𝗰𝗮𝗰𝗵𝗲: Redis, Memcached, or in-memory stores between the app and database. Reduces database load for hot-path lookups like session data, catalogs, or feature flags. → 𝗖𝗗𝗡 / 𝗲𝗱𝗴𝗲 𝗰𝗮𝗰𝗵𝗲: Content served from locations closer to the user. Relevant for serving dashboards and reports at scale. → 𝗦𝗽𝗮𝗿𝗸 𝗰𝗮𝗰𝗵𝗲: .cache() or .persist() to keep intermediate DataFrames in memory across stages. Avoids recomputing the same transformation in multi-step pipelines. 𝗧𝗵𝗲 𝘁𝗿𝗮𝗱𝗲-𝗼𝗳𝗳 𝘁𝗵𝗮𝘁 𝗻𝗲𝘃𝗲𝗿 𝗰𝗵𝗮𝗻𝗴𝗲𝘀: Speed vs staleness. Every cache is a snapshot of a past state. The faster it serves, the more likely it's serving data that's no longer current. Cache invalidation remains one of the hardest problems not because it's complex to code, but because it's complex to get right under changing data. 𝗧𝗵𝗲 𝗱𝗲𝗰𝗶𝘀𝗶𝗼𝗻 𝗿𝘂𝗹𝗲: Cache what's expensive to compute, frequently accessed, and tolerant of slight staleness. If freshness is non-negotiable, caching is a liability not a shortcut. Where in your stack are you trading freshness for speed and is the trade-off still worth it? #DataEngineering #DataArchitecture #SystemDesign

I was asked in an interview “"Where can we cache data apart from the DB layer?” Caching helps store frequently accessed or computationally expensive data closer to where it's needed — reducing response time and improving scalability. It is not just about saving DB hits, but about optimizing latency and load throughout the entire stack. While it's common to place a cache near the database (e.g., Redis/Memcached), here are other layers where caching can be just as powerful: - Client devices – Cache API responses, UI state, and static assets in LocalStorage on client side - CDN – Cache static files (images, JS, CSS) and public GET API responses at edge locations - API Gateway – Cache GET endpoint responses or auth metadata to offload traffic from services - Load Balancers – Cache routing metadata or session affinity information for efficient request distribution - Web application servers – Cache user profiles, computed business logic, or results from third-party APIs in memory or a distributed cache Caching decisions vary by use case, but knowing where and what to cache can make a significant difference in system performance at scale. #SystemDesign #SoftwareEngineering #Caching #Scalability #DistributedSystems

Uber's database handles 40M reads/s. 4.1 ms at p99.9 latency explained: Uber's Docstore is a massive, distributed database essential for handling the company's data needs, processing more than 30 million requests per second. As demand grew, the need for low-latency, high-throughput solutions became critical. Traditional disk-based storage, even with optimizations like NVMe SSDs, faced limitations in scalability, cost, and latency. To address these challenges, Uber developed CacheFront, an integrated caching solution designed to reduce latency, improve scalability, and lower costs without compromising on data consistency or developer productivity. The Key Challenges: 0. Latency and scalability: Disk-based databases have inherent latency and scalability limits. 1. Cost: Scaling up or horizontally adds significant costs. 2. Operational complexity: Managing increased partitions and ensuring data durability is complex. 3. Request imbalance: High-read request volumes can overwhelm storage nodes. CacheFront's solution had to implement cached reads, with disk fallback if necessary. It also had to handle cache invalidation with a data capture system and remain adaptable per database, table, or request basis. Implementation Highlights: 0. Incremental build: Started with the most common query patterns for caching. 1. High-level architecture: Separates caching from storage, allowing independent scaling. 2. Cache invalidation: Utilizes change data capture to maintain consistency. 3. Cache warming and sharding: Ensures high availability and fault tolerance across geographical regions. 4. Circuit breakers and adaptive timeouts: Enhances system resilience and optimizes latency. Results and Impact: 0. P75 latency decreased by 75%, and P99.9 latency by over 67%. 1. Achieved a 99% cache hit rate for some of the largest use cases, significantly reducing the load on the storage engine. 2. Reduced the need for approximately 60K CPU cores to just 3K Redis cores for certain use cases. 3. Supports over 40 million requests per second across all instances, with proven success in failover scenarios. Actionable Learnings: 0. Integrated caching: Implementing an integrated caching layer can dramatically improve database read performance while reducing costs and operational complexity. 1. Cache invalidation: A robust cache invalidation strategy is crucial for maintaining data consistency, especially in systems requiring high throughput and low-latency reads. 2. Adaptability and scalability: Systems should be designed to adapt to varying workloads and scale independently across different components to ensure reliability and performance. What do you think about it? #softwareengineering #systemdesign #programming

𝗖𝗮𝗰𝗵𝗶𝗻𝗴 𝘃𝘀 𝗡𝗼 𝗖𝗮𝗰𝗵𝗶𝗻𝗴: 𝗪𝗵𝘆 𝗶𝗻𝘁𝗲𝗹𝗹𝗶𝗴𝗲𝗻𝘁 𝗰𝗮𝗰𝗵𝗶𝗻𝗴 𝗶𝘀 𝗶𝗺𝗽𝗼𝗿𝘁𝗮𝗻𝘁 𝗳𝗼𝗿 𝗔𝗜 𝘄𝗼𝗿𝗸𝗳𝗹𝗼𝘄𝘀 Ever wondered how much money you're wasting on repeated AI queries? I built a real experiment with Agno and AgentOps to find out. 𝗧𝗵𝗲 𝗦𝗲𝘁𝘂𝗽: I created a workflow caching demo using Agno: First execution: Cache miss → hits OpenAI API Second execution: Cache hit → instant response from memory Same exact query: "Tell me a joke" 𝗕𝘂𝗶𝗹𝗱𝗶𝗻𝗴 𝗦𝗺𝗮𝗿𝘁 𝗖𝗮𝗰𝗵𝗶𝗻𝗴 𝗶𝘀 𝗘𝗔𝗦𝗬: With Agno, you just: Use `self.session_state[message]` = content to cache Check if `self.session_state.get(message):` for hits yield from `self.agent.run()` for streaming Agno's built-in session state IS your cache. No Redis, no complexity. 𝗢𝗯𝘀𝗲𝗿𝘃𝗮𝗯𝗶𝗹𝗶𝘁𝘆 𝘄𝗶𝘁𝗵 𝗔𝗴𝗲𝗻𝘁𝗢𝗽𝘀: Just `agentops.init()` and I tracked: - Cache hit vs miss patterns - Exact response times - Cost breakdown per operation - Session state evolution - Two separate traces showing completely different behavior patterns. 𝗧𝗵𝗲 𝗥𝗲𝘀𝘂𝗹𝘁𝘀: Cache Miss: 𝟭.𝟮 𝘀𝗲𝗰𝗼𝗻𝗱𝘀, $𝟬.𝟬𝟮 Cache Hit: 𝟬.𝟬𝟬𝟭 𝘀𝗲𝗰𝗼𝗻𝗱𝘀, $𝟬.𝟬𝟬 That's 𝟵𝟵.𝟵% 𝗳𝗮𝘀𝘁𝗲𝗿 and 𝟭𝟬𝟬% 𝗰𝗼𝘀𝘁 𝘀𝗮𝘃𝗶𝗻𝗴𝘀 for repeated queries! 𝗪𝗵𝘆 𝗧𝗵𝗶𝘀 𝗠𝗮𝘁𝘁𝗲𝗿𝘀: FAQ systems save 100% on repeated questions Development cycles speed up dramatically Production costs plummet for common queries User experience becomes instant 𝗧𝗵𝗲 𝗧𝗮𝗸𝗲𝗮𝘄𝗮𝘆: Stop paying for the same AI responses twice. Build intelligent caching with Agno, and use AgentOps to prove your optimizations work. Most frameworks make caching complex. Agno makes it feel like regular Python. It's saving me hundreds on API costs already! What's your biggest AI cost optimization win? Drop it in the comments! 👇 Agency

No Caching = Performance Bottleneck One of the most overlooked cloud performance antipatterns is not caching data at all. You’d be surprised how many systems fetch the same data repeatedly—despite it rarely changing. Here’s what happens when you fall into the No Caching Antipattern: 🔁 Repeated DB queries for identical data 🐌 Slow response times under load 🔥 Increased I/O, latency, and cloud costs ⛔️ Risk of service throttling or failure ✅ The Fix? Cache-Aside Pattern 1. Try to get from cache 2. If not found, fetch from DB + store in cache 3. Invalidate or update on write How to Detect the No Caching Antipattern 🔍 Review app design: Is any cache layer used? Which data changes slowly? 📊 Instrument the system: How often are the same requests made? 🧪 Profile the app: Check I/O, CPU, and memory usage in a test environment 🚦 Load test: Simulate realistic workloads to measure impact under stress 📈 Analyze DB/query stats: Which queries are repeated the most? Tip: Even if data is volatile or short-lived, smart caching strategies (with TTL, invalidation, and fallbacks) can massively improve resilience and scalability. Cache wisely. Profile constantly. Monitor cache hit rates. Because not caching is costing you more than you think. Have you encountered this in the wild? Drop your experience below 👇

If your agent doesn't cache, it's burning budget! Traditional LLM queries behaves predictably: you ask once, wait briefly, and get a clear answer. But AI agents think differently - they plan, retry, call external tools, and loop through multiple reasoning steps. This can quickly blow up latency and costs. Here's the three ways caching saves your AI agents from cost meltdown. 1. Mem caching: Keep a simple in-session storage that remembers what your agent already asked or computed, preventing duplicated calls. It holds intermediate tool results, retrieved snippets, or reasoning steps so the agent doesn’t redo work inside the same task. 2. Prefix caching: Break agent answers into chunks. Save reusable pieces so subsequent queries just extend from a known prefix. If the agent already generated the first 400 tokens of a long answer, you can resume from there instead of re‑prompting the whole sequence. 3. Prompt caching: Set up a basic store that maps frequent prompts directly to answers, skipping repeated computation for identical questions. Great for FAQs and idempotent calls; useless when the context changes per request. Caching is about keeping your agents responsive and your costs predictable. It’s how you turn clever loops into predictable, affordable workflows. Keep shining a light on it - most teams still overlook the basics. #FinOps #Mavvrik #AgenticAI

When RAG isn’t enough and CAG becomes essential Imagine you’re building a voice assistant for aircraft maintenance engineers. Each aircraft type has tens of thousands of validated maintenance procedures, torque settings, inspection steps, fault codes stored in secure internal manuals that rarely change. At first glance, you might think of using Retrieval-Augmented Generation (RAG). Every time an engineer asks, “What’s the torque setting for the A320 hydraulic pump valve?” the system would: Retrieve relevant text from thousands of manuals, Rank and embed them, Then feed that data into the LLM for response generation. This works beautifully when knowledge is dynamic and expansive ,but in this case, it introduces unnecessary latency, indexing dependencies, and retrieval noise, even though the knowledge is largely static and validated. Enter Cache-Augmented Generation (CAG) Now, imagine replacing those repeated retrieval calls with a cached knowledge layer. With CAG, frequently used and stable information, like standard procedures or torque settings ,is pre-encoded and cached in a key-value memory or persistent context store. When the engineer asks a question, the LLM doesn’t perform an external retrieval. It fetches from the cache, instantly pulling the pre-validated answer context into generation. Why CAG makes sense here The domain is bounded and stable , knowledge doesn’t change often Speed and reliability matter more than continuous updates Only a high value subset of knowledge needs caching, not all 50,000 documents The system still supports retrieval as fallback for new or un-cached data So, the assistant now behaves intelligently: It fetches known procedures directly from cache, It retrieves only when something new or unknown is asked. My take : Hybrid is the future No enterprise runs purely on CAG or RAG alone. Modern architectures use adaptive hybrids , cache for repeatable, validated knowledge; retrieval for evolving information. This hybridization reduces latency, maintains freshness, and provides deterministic accuracy where it matters most. #RAG #CAG #AI #LLM's Image credit Mohamad Hassan, unable to tag him.

You can reduce latency by >2x and costs up to 90% with "prompt caching". Prompt Caching enables caching of frequently used context between API calls — resulting with shorter response times, and lower processing costs. But how does it work? When an LLM processes a prompt, it generates internal representations called attention states, which help the model understand relationships between different parts of the input. Traditionally, these attention states are recalculated every time the model processes a prompt, even if the input is similar or repeated, making it time-consuming and costly. Prompt caching solves this by storing previously computed attention states, allowing the model to retrieve them for similar prompts, speeding up responses and reducing costs. So, when should you use it? Prompt caching is useful for sending large prompt context once and reusing it in future requests, reducing cost and latency. 1/ In conversational agents, it enhances extended dialogues by avoiding repeated input of long instructions. 2/ For coding assistants, it helps by keeping a summarized version of the codebase in the prompt for faster autocomplete and Q&A. 3/ In large document processing, it allows full documents or images to be included without slowing down response times. 4/ It also improves the quality of responses in detailed instruction sets and agentic search by maintaining a cache of multiple examples and iterative tool calls. You're probably using this feature automatically with OpenAI models, and you can set it up manually for Anthropic and Gemini models. Check our parameter library for more info: https://lnkd.in/dczSJ6yA