Preventing Context Window Waste in AI Workflows

You are paying for billions of tokens each day before generating a single useful output 💸 At Twitter, we cut ads ranking prediction costs by 85% - not with a better model, but by fixing payload bloat. The same pattern is showing up again with MCP. It’s brilliant for developer workflows, but naive production deployments create a “context-window tax” that compounds silently. Here's the math people aren't doing: → ~3,000 tokens of tool/schema context per request → 500k daily requests → billions of tokens/day Yes, caching helps - a lot. But only if prompts are structured for reuse. Most aren’t. Here are the top 4 things to solve this architecture problem: ❶ Default to cheap routers. Regex, embeddings, small fine-tuned models, or at most Flash/Haiku/nano-tier LLMs. Frontier models should be the last resort. The cost delta is 3–5x with negligible routing quality difference! ❷ Decouple orchestration from reasoning. Lightweight models handle tool use & APIs. Frontier models handle synthesis, multi-step reasoning, and ambiguity. Don’t use a sledgehammer to sort mail. ❸ Treat context like a production resource. Don’t inject every tool schema into every request. Scope tools, compress schemas, and load lazily. Every token costs on every call. ❹ Cache aggressively, but correctly. Prompt caching can cut costs up to 90% (Anthropic, OpenAI, Google DeepMind). But it only works if prefixes are stable and prompts are reusable. The best ML systems aren't the most clever. They're the ones that minimize tokens, isolate expensive reasoning, and make cost-quality tradeoffs explicit. This is Part 1 of my MCP production teardown. Over the next few weeks, I’ll share insights on Shadow AI protocols, model-agnosticism, memory vs reflex, and more. If you're building Gen AI systems at scale, I’d love to hear from you. Curious what’s been your highest cost or latency bottleneck so far.

One of the biggest challenges I see with scaling LLM agents isn’t the model itself. It’s context. Agents break down not because they “can’t think” but because they lose track of what’s happened, what’s been decided, and why. Here’s the pattern I notice: 👉 For short tasks, things work fine. The agent remembers the conversation so far, does its subtasks, and pulls everything together reliably. 👉 But the moment the task gets longer, the context window fills up, and the agent starts forgetting key decisions. That’s when results become inconsistent, and trust breaks down. That’s where Context Engineering comes in. 🔑 Principle 1: Share Full Context, Not Just Results Reliability starts with transparency. If an agent only shares the final outputs of subtasks, the decision-making trail is lost. That makes it impossible to debug or reproduce. You need the full trace, not just the answer. 🔑 Principle 2: Every Action Is an Implicit Decision Every step in a workflow isn’t just “doing the work”, it’s making a decision. And if those decisions conflict because context was lost along the way, you end up with unreliable results. ✨ The Solution to this is "Engineer Smarter Context" It’s not about dumping more history into the next step. It’s about carrying forward the right pieces of context: → Summarize the messy details into something digestible. → Keep the key decisions and turning points visible. → Drop the noise that doesn’t matter. When you do this well, agents can finally handle longer, more complex workflows without falling apart. Reliability doesn’t come from bigger context windows. It comes from smarter context windows. 〰️〰️〰️ Follow me (Aishwarya Srinivasan) for more AI insight and subscribe to my Substack to find more in-depth blogs and weekly updates in AI: https://lnkd.in/dpBNr6Jg

Something i learnt the hard way....Bigger context window ≠ smarter AI agent. In fact, it often makes it worse. I've seen this mistake in multiple builds. Team upgrades the model. Gets a massive context window. Then starts dumping everything into it. Logs. History. Documents. Old prompts. Feels powerful. Until the agent starts behaving... strangely. Let us see what's actually happening. This is something we don't talk about enough: Context Rot The model isn't thinking more. It's just gets distracted. It spends its attention on irrelevant noise and falls back to repeating past patterns instead of reasoning fresh. Context Poisoning ..is even worse. One slightly wrong piece of information sneaks in and now every downstream step builds on it. Silently. That's when production systems start giving confidently wrong outputs. Here's the ground reality... Memory is not a model problem. It's a systems design problem. If you're building agentic workflows, this is where things usually break and how to fix them 1. RAG is NOT memory: RAG is like a reference. Static. Read-only. But agent memory?? Needs to evolve, needs to write back, needs to remember user-specific context over time. If your system can't do that, it's not memory. It's lookup. 2. If you don't forget, you will fail: Most teams design memory like a warehouse. Store everything. Bad idea. what you need is TTL (time-based eviction), Recency weighting, Active cleanup. Otherwise your agent starts reasoning on outdated information. 3. Similarity alone is not enough: Most pipelines rely only on semantic similarity. But in real systems relevance = similarity + recency + importance. Without this, important details surface over critical facts. And the agent drifts. 4. The weird one (but real): This surprised even me. When context is perfectly structured, models sometimes get distracted. But when information is slightly shuffled. They actually retrieve facts better. Not intuitive. But shows how fragile attention really is. We treat context window like storage. Actually it's not. It's a scarce cognitive resource. Don't let your agents hoard data. Give them space to think. Like to know how your are solving this. Are you building custom memory systems or relying on frameworks like LangGraph / Redis and adapting them?

Context Engineering: an essential new skill. Perhaps you've spent a year learning to prompt better. Asking the right questions, in the right way, with the right tone. It's made us all more effective at casual AI use. Now we need to turn these ideas into systems that can take on every-more-complex taskwork. We need to go beyond prompt engineering. But when you try to deploy AI for anything serious like bid writing, design reviews, technical reports, prompt engineering alone will not suffice. Because the problem isn't how you're asking the question. The problems is exactly what information the AI has access to when it tries to answer. This is context engineering: the deliberate curation and sequencing of data you feed into an AI system. Context window is finite, so we must use it strategically to reach our end goal. Consider a typical bid-writing response. You need relevant case studies, specific client research, the right CVs, past winning answers, and your practice voice. That could easily be 3 million tokens of data. But the largest context window available is around 1 million tokens. You physically cannot dump everything in and press go. Context engineering forces you to be strategic. Build a research agent to gather client intelligence. Create a knowledge agent loaded with your best project case studies. Deploy a database agent that can pull relevant CVs on demand. Orchestrate them in sequence, each adding critical information to the feed, so the final output is genuinely fit for purpose. Pick one critical process, draw it out on paper, and work out what information needs to flow where and when.

I burned $47 in API calls watching an agent check the same API endpoint every 30 seconds for 6 hours straight. It was supposed to monitor a deployment status. Instead, it just… kept checking. No breaks. No strategy. Just pure, expensive anxiety. That's the problem with current AI agents: they're brilliant at one-time tasks but absolutely terrible at waiting. Microsoft Research just released something that fixes this: SentinelStep. It's now open-sourced in their Magentic-UI system, and honestly, this changes how we think about agent workflows. Here's what makes it work: The system breaks monitoring into three components: actions (what to check), conditions (when to stop), and polling intervals (how often to check). Simple concept, but the execution is clever. Dynamic polling is where it gets interesting. The agent doesn't blindly check every minute. It makes an educated guess based on task urgency. Monitoring quarterly earnings? Less frequent checks. Tracking an urgent email? More aggressive polling. Then it adjusts based on observed patterns. Now, here's my take on what's probably happening behind the scenes: The system likely maintains a state snapshot after the first check, basically freezing what the agent knows at that moment. Think of it like taking a photo of the agent's brain. For each subsequent check, instead of carrying forward the entire conversation history (which would expand the context window), it loads the frozen snapshot, performs the new check, compares the results, and determines whether the condition is met. The polling adjustment probably uses something straightforward maybe exponential backoff with task-specific multipliers. If nothing changes after a few checks, wait longer next time. If patterns emerge (like "emails usually arrive between 9-11 AM"), the interval shrinks during those windows. No fancy ML needed, just sensible heuristics. Context management is the real win here. Without it, a 2-day monitoring task would accumulate thousands of tokens of redundant checks. With state snapshots, each check stays isolated and lightweight. They tested it with SentinelBench and showed success rates jumping from 5.6% to 33-39% for 1-2 hour tasks. But here's what I think matters more than those numbers: where you'd actually use this. Imagine monitoring CI/CD pipelines that take hours to complete, tracking competitor pricing that updates sporadically, or watching for specific social media mentions across days. These aren't hypothetical—they're tasks we currently handle with clunky cron jobs or manual checking. pip install magentic-ui right now and start experimenting. The foundation is solid, though you'll want to test thoroughly for production use cases (Microsoft's transparency note calls this out explicitly). This feels like one of those unglamorous infrastructure pieces that quietly enable a whole new category of automation. Not flashy, but exactly what we needed.

𝗔𝗜-𝗔𝘀𝘀𝗶𝘀𝘁𝗲𝗱 𝗖𝗠: 𝗙𝗿𝗼𝗺 𝗖𝗼𝗻𝘁𝗲𝘅𝘁 𝗥𝗼𝘁 𝘁𝗼 𝗥𝗶𝗴𝗼𝗿𝗼𝘂𝘀 𝗦𝗰𝗮𝗳𝗳𝗼𝗹𝗱𝗶𝗻𝗴 Your AI-assisted product change starts brilliantly. The first analysis is excellent, and the second builds reasonably well. By the fourth interaction, the AI contradicts earlier decisions and forgets critical constraints. This isn't AI failure; it's context degradation. Large language models have fixed context windows. As conversation accumulates, earlier exchanges compress or disappear. The scaffolding pattern, as demonstrated by Benedict Smith, addresses this through structured techniques mapping directly to CM governance. 𝗖𝗼𝗻𝘁𝗲𝘅𝘁 𝗲𝗻𝗴𝗶𝗻𝗲𝗲𝗿𝗶𝗻𝗴 maintains structured project files providing consistent information to each AI interaction. Effective implementations use explicit configuration state documents capturing scope, affected components, constraints, and design intent. This is standard change control documentation. Organizations maintaining rigorous CM baselines already have this discipline. 𝗧𝗮𝘀𝗸 𝗱𝗲𝗰𝗼𝗺𝗽𝗼𝘀𝗶𝘁𝗶𝗼𝗻 breaks workflows into atomic, verifiable units. Instead of "complete this change," decompose it into discrete tasks: generate CAD modifications, run FMEA, and validate BOM consistency, each as a separate interaction with clear acceptance criteria. 𝗦𝘂𝗯-𝗮𝗴𝗲𝗻𝘁 𝗲𝘅𝗲𝗰𝘂𝘁𝗶𝗼𝗻 deliberately discards context between tasks. Each discrete task executes in a fresh AI instance with only relevant context, preventing error propagation. According to research on context degradation, effective context windows are "much smaller than advertised token limits." This phenomenon—"context rot"—means LLM performance degrades as the context window fills, making scaffolding essential. Scaffolding aligns with governance requirements. Organizations maintaining rigorous CM2 baselines, clear change processes, and structured documentation already have what scaffolding requires. PLM systems should become an infrastructure that scaffolded workflows interact with, not monolithic interfaces that engineers navigate manually. Context files maintained in version control capture design intent. Validation agents enforce constraints automatically. Human approval gates preserve accountability. One could start with scaffolding for specific, bounded workflows where governance requirements are well-understood. Engineering change orders affecting well-characterized part families. Build expertise where failure is recoverable before extending to safety-critical applications. If your team can't explicitly articulate CM requirements for structured prompting, do they lack the discipline needed to manage CM effectively, even without automation? What's your experience with sustained AI workflows? Have you encountered context degradation in multi-day configuration management tasks? #AI #CM2 #ConfigurationManagement #PLM #ProductLifecycleManagement #CM #IpX #MDUX

Context Engineering is the #1 Gen AI Skill in 2026 For 99% people, that means using 'Projects' properly LLMs like ChatGPT and Claude have this powerful feature called Projects However, most NEVER set up project correctly - leading to more AI slop First thing, remember every project has 3 components ↳ Memory ↳ Instructions ↳ Files 1. Separate Instructions from Knowledge (when setting up Project) ↳ Instructions: HOW to behave (tone, format) ↳ Files: WHAT to know (uploaded files) Example: Upload 'Style Guide' or 'Design System' as a file in Files, not as an Instruction 2. Be Explicit in Instructions ↳ Don't write vague instructions like 'Be helpful and organized' ↳ Use explicit structure such as 'Problem - Why It Matters - Solution - Example' In my usage and testing, I have found that Claude 4.x and GPT-5.x follow instructions quite literally 3. Upload Files, Don't Paste in conversations ↳ Stop copy-pasting brand guidelines into every chat ↳ Upload once: PDFs, Docs, spreadsheets, images and AI references automatically ↳ Then reference them, instead of re-explaining them every time 4. Use Markdown Structure (get help from ChatGPT) ↳ Headers, bullets, code blocks beat an unformatted blob of text ↳ Models read structured formatting more reliably 5. Name Files Descriptively ↳ Don't upload files with names like 'Document1.pdf' ↳ Name it appropriately such as 'Brand_Voice_Guidelines_2026.pdf' ↳ Filenames help AI retrieve the right context 6. Keep Knowledge Base Lean ↳ More files ≠ better performance. ↳ Too much context leads to performance degradation ↳ Only upload what's essential for THIS project 7. Isolate by Context Boundary ↳ Treat each project as a separate memory bubble ↳ Client A work stays separate from Client B ↳ Confidential work should not overlap into client-facing projects 8. Add Examples for Complex Tasks (few-shot) ↳ Simple tasks: skip examples ↳ Specific workflows: show 1-2 good quality sample exchanges 9. Audit Monthly ↳ Memory: ChatGPT default: viewable/editable overall for user ChatGPT project-only: BLACK BOX Claude: transparent, editable summaries for each Project ↳ Files: Delete outdated docs, old team members, expired info Stale context leads to degraded performance I, myself, break down projects like this: ↳ one Project per client ↳ one for PM work - experiments, prototypes ↳ one for Strategy ↳ one for LinkedIn posts ↳ one for the newsletter No matter what work you do - whether it is strategy, writing, research, media, content, documentation - your LLM is only as good as the context you give it. --------- I am Priyadeep Sinha and I help you level up with AI one strategy at a time For my best, most detailed resources, subscribe to my weekly newsletter Work in Beta where I share the AI strategies I used as a VP-Product building winning AI products: https://lnkd.in/gPqYEzaJ

Anthropic just posted another banger guide. This one is on building more efficient agents to handle more tools and efficient token usage. This is a must-read for AI devs! (bookmark it) It helps with three major issues in AI agent tool calling: token costs, latency, and tool composition. How? It combines code executions with MCP, where it turns MCP servers into code APIs rather than direct tool calls. Here is all you need to know: 1. Token Efficiency Problem: Loading all MCP tool definitions upfront and passing intermediate results through the context window creates massive token overhead, sometimes 150,000+ tokens for complex multi-tool workflows. 2. Code-as-API Approach: Instead of direct tool calls, present MCP servers as code APIs (e.g., TypeScript modules) that agents can import and call programmatically, reducing the example workflow from 150k to 2k tokens (98.7% savings). 3. Progressive Tool Discovery: Use filesystem exploration or search_tools functions to load only the tool definitions needed for the current task, rather than loading everything upfront into context. This solves so many context rot and token overload problems. 4. In-Environment Data Processing: Filter, transform, and aggregate data within the code execution environment before passing results to the model. E.g., filter 10,000 spreadsheet rows down to 5 relevant ones. 5. Better Control Flow: Implement loops, conditionals, and error handling with native code constructs rather than chaining individual tool calls through the agent, reducing latency and token consumption. 6. Privacy: Sensitive data can flow through workflows without entering the model's context; only explicitly logged/returned values are visible, with optional automatic PII tokenization. 7. State Persistence: Agents can save intermediate results to files and resume work later, enabling long-running tasks and incremental progress tracking. 8. Reusable Skills: Agents can save working code as reusable functions (with SKILL .MD documentation), building a library of higher-level capabilities over time. This approach is complex and it's not perfect, but it should enhance the efficiency and accuracy of your AI agents across the board. anthropic. com/engineering/code-execution-with-mcp

I used this guide to build 10+ AI Agents Here're my 10 actionable items: 1. Turn your agent into a note-taking machine → Dump plans, decisions, and results into state objects outside the context window → Use scratchpad files or runtime state that persists during sessions → Stop cramming everything into messages - treat state like external storage 2. Be ridiculously picky about what gets into context → Use embeddings to grab only memories that matter for current tasks → Keep simple rules files (like CLAUDE md) that always load → Filter tool descriptions with RAG so agents aren't confused by irrelevant tools 3. Build a memory system that remembers useful stuff → Create semantic, episodic, and procedural memory buckets for facts, experiences, instructions → Use knowledge graphs when embeddings fail for relationship-based retrieval → Avoid ChatGPT's mistake of pulling random location data into unrelated requests 4. Compress like your context window costs $1000 per token → Set auto-summarization at 95% context capacity with no exceptions → Trim old messages with simple heuristics: keep recent, dump middle → Post-process heavy tool outputs immediately - search results don't live forever 5. Split your agent into specialized mini-agents → Give each sub-agent one job and its own isolated context window → Hand off context with quick summaries, not full message histories → Run sub-agents in parallel when possible for isolated exploration 6. Sandbox the heavy stuff away from your LLM → Execute code in environments that isolate objects from context → Store images, files, complex data outside the context window → Only pull summary info back - full objects stay in sandbox 7. Make summarization smart, not just chronological → Train models specifically for agent context compression → Preserve critical decision points while compressing routine chatter → Use different strategies for conversations vs tool outputs 8. Prune context like you're editing a novel → Implement trained pruners that understand relevance, not just recency → Filter based on task relevance while maintaining conversational flow → Adjust pruning aggressiveness based on task complexity 9. Monitor token usage like a hawk → Track exactly where tokens burn in your agent pipeline → Set real-time alerts when context utilization hits dangerous levels → Build dashboards correlating context management with success rates 10. Test everything or admit you're just guessing → A/B test different context strategies and measure performance differences → Create evaluation frameworks testing before/after context engineering changes → Set up continuous feedback loops auto-adjusting context parameters Last but not the least, be open to new ideas and keep learning Check out 50+ AI Agent Tutorials on my profile 👋 .

You can’t 𝐛𝐮𝐢𝐥𝐝 𝐫𝐞𝐥𝐢𝐚𝐛𝐥𝐞 𝐀𝐈 𝐚𝐠𝐞𝐧𝐭𝐬 without mastering this core skill: 𝐜𝐨𝐧𝐭𝐞𝐱𝐭 𝐞𝐧𝐠𝐢𝐧𝐞𝐞𝐫𝐢𝐧𝐠. → Here’s the full picture in one visual. 👇 Here’s what’s really going on when you want your agent to reason, retrieve, interact with tools, and stay efficient over time: 🔹 𝐂𝐨𝐧𝐭𝐞𝐱𝐭 𝐞𝐧𝐠𝐢𝐧𝐞𝐞𝐫𝐢𝐧𝐠 is the process of selecting and feeding the right information into the LLM’s context window. It guides the model’s output during reasoning or task execution. It includes: → 𝐈𝐧𝐬𝐭𝐫𝐮𝐜𝐭𝐢𝐨𝐧𝐬 (prompts, few-shot, tool specs) → 𝐊𝐧𝐨𝐰𝐥𝐞𝐝𝐠𝐞 (facts, memories) → 𝐓𝐨𝐨𝐥𝐬 (feedback from tool calls) 🔍 When applied to AI Agents, it solves a major 𝐩𝐚𝐢𝐧 𝐩𝐨𝐢𝐧𝐭: > Repeated LLM + tool interleaving in long tasks = large token usage ⚠️ Token overload leads to: → Exceeding context window → Increased latency and cost → Poorer agent reasoning 🔹 To manage that, here are 4 context engineering 𝐬𝐭𝐫𝐚𝐭𝐞𝐠𝐢𝐞𝐬: 1. 𝐖𝐫𝐢𝐭𝐞 𝐜𝐨𝐧𝐭𝐞𝐱𝐭 (scratchpads, memories, tool call state) 2. 𝐒𝐞𝐥𝐞𝐜𝐭 𝐜𝐨𝐧𝐭𝐞𝐱𝐭 (choose what to load using RAG, memory types) 3. 𝐂𝐨𝐦𝐩𝐫𝐞𝐬𝐬 𝐜𝐨𝐧𝐭𝐞𝐱𝐭 (summarization, trimming) 4. 𝐈𝐬𝐨𝐥𝐚𝐭𝐞 𝐜𝐨𝐧𝐭𝐞𝐱𝐭 (multi-agent systems) 🧩 It's not just about stuffing more into a prompt—it’s about orchestrating memory, retrieval, and structure across the agent's workflow. Source: 👇