How to Reduce Generative AI Model Costs

You are paying for billions of tokens each day before generating a single useful output 💸 At Twitter, we cut ads ranking prediction costs by 85% - not with a better model, but by fixing payload bloat. The same pattern is showing up again with MCP. It’s brilliant for developer workflows, but naive production deployments create a “context-window tax” that compounds silently. Here's the math people aren't doing: → ~3,000 tokens of tool/schema context per request → 500k daily requests → billions of tokens/day Yes, caching helps - a lot. But only if prompts are structured for reuse. Most aren’t. Here are the top 4 things to solve this architecture problem: ❶ Default to cheap routers. Regex, embeddings, small fine-tuned models, or at most Flash/Haiku/nano-tier LLMs. Frontier models should be the last resort. The cost delta is 3–5x with negligible routing quality difference! ❷ Decouple orchestration from reasoning. Lightweight models handle tool use & APIs. Frontier models handle synthesis, multi-step reasoning, and ambiguity. Don’t use a sledgehammer to sort mail. ❸ Treat context like a production resource. Don’t inject every tool schema into every request. Scope tools, compress schemas, and load lazily. Every token costs on every call. ❹ Cache aggressively, but correctly. Prompt caching can cut costs up to 90% (Anthropic, OpenAI, Google DeepMind). But it only works if prefixes are stable and prompts are reusable. The best ML systems aren't the most clever. They're the ones that minimize tokens, isolate expensive reasoning, and make cost-quality tradeoffs explicit. This is Part 1 of my MCP production teardown. Over the next few weeks, I’ll share insights on Shadow AI protocols, model-agnosticism, memory vs reflex, and more. If you're building Gen AI systems at scale, I’d love to hear from you. Curious what’s been your highest cost or latency bottleneck so far.

After optimizing costs for many AI systems, I've developed a systematic approach that consistently delivers cost reductions of 60-80%. Here's my playbook, in order of least to most effort: Step 1: Optimizing Inference Throughput Start here for the biggest wins with least effort. Enabling caching (LiteLLM (YC W23), Zilliz) and strategic batch processing can reduce costs by a lot with very little effort. I have seen teams cut costs by half simply by implementing caching and batching requests that don't require real-time results. Step 2: Maximizing Token Efficiency This can give you an additional 50% cost savings. Prompt engineering, automated compression (ScaleDown), and structured outputs can cut token usage without sacrificing quality. Small changes in how you craft prompts can lead to massive savings at scale. Step 3: Model Orchestration Use routers and cascades to send prompts to the cheapest and most effective model for that prompt (OpenRouter, Martian). Why use GPT-4 for simple classification when GPT-3.5 will do? Smart routing ensures you're not overpaying for intelligence you don't need. Step 4: Self-Hosting I only suggest self-hosting for teams at scale because of the complexities involved. This requires more technical investment upfront but pays dividends for high-volume applications. The key is tackling these layers systematically. Most teams jump straight to self-hosting or model switching, but the real savings come from optimizing throughput and token efficiency first. What's your experience with AI cost optimization?

Are you using any draft-first or adaptive reasoning strategies in production? AI models are overthinking. And it's costing us. Most LLMs use chain-of-thought reasoning — writing out every intermediate step before answering. It works. But it's slow, expensive, and token-heavy. What if we trained models to reason with only the tokens they actually need? The approach uses a two-stage RL pipeline: → Stage 1: Reward the model for being concise. → Stage 2: Add an accuracy reward so it doesn't just become terse and wrong. The combined reward looks like this: R = λ_eff · R_eff + λ_acc · R_acc The model learns to find the minimum reasoning path that still gets the right answer. Results: ✦ 25–30% fewer tokens. ✦ Accuracy stayed the same or slightly improved. ✦ Works across GPT-4, Llama, Claude no model-specific tuning needed. The practical implications are real: ~ 30% lower inference costs at scale. ~ Faster responses for latency-critical apps. ~ Shorter traces that finally make on-device LLMs viable. ~ Smarter routing: draft reasoning for easy queries, full CoT only when it's hard. The trade-off: The RL fine-tuning costs GPU hours upfront. But for any high-volume service, that's a one-time investment that pays back on every single inference. The deeper insight here isn't just about efficiency. It's that models don't need to show all their work to be right. Just enough of it. #DraftThinking

Most engineers think model cost is about API tokens or inference time.  In reality, it’s about how your requests compete for GPU scheduling and how effectively your data stays hot in cache. Here’s the untold truth 👇 1. 𝐄𝐯𝐞𝐫𝐲 𝐦𝐢𝐥𝐥𝐢𝐬𝐞𝐜𝐨𝐧𝐝 𝐨𝐧 𝐚 𝐆𝐏𝐔 𝐢𝐬 𝐚 𝐰𝐚𝐫 𝐟𝐨𝐫 𝐩𝐫𝐢𝐨𝐫𝐢𝐭𝐲. .   Your model doesn’t just “run.” It waits its turn.   Schedulers (like Kubernetes device plugins, Triton schedulers, or CUDA MPS) decide who gets compute time — and how often.   If your jobs are fragmented or unbatched, you’re paying for idle silicon.   That’s like renting a Ferrari to sit in traffic. 2. 𝐂𝐚𝐜𝐡𝐢𝐧𝐠 𝐥𝐚𝐲𝐞𝐫𝐬 𝐪𝐮𝐢𝐞𝐭𝐥𝐲 𝐝𝐞𝐜𝐢𝐝𝐞 𝐲𝐨𝐮𝐫 𝐛𝐮𝐫𝐧 𝐫𝐚𝐭𝐞.   Intermediate activations, embeddings, and KV caches live in high-bandwidth memory.   If your model keeps reloading them between requests — you’re paying full price every time.   That’s why serving infra (like vLLM, DeepSpeed, or FasterTransformer) focuses more on cache reuse than raw FLOPS. The real optimization isn’t in “faster models.”   It’s in smarter scheduling and cache locality.   Your cost per token can drop 50% with zero model changes — just better orchestration. 3. 𝐓𝐡𝐞 𝐡𝐢𝐝𝐝𝐞𝐧 𝐭𝐚𝐱: 𝐟𝐫𝐚𝐠𝐦𝐞𝐧𝐭𝐚𝐭𝐢𝐨𝐧 𝐚𝐧𝐝 𝐞𝐯𝐢𝐜𝐭𝐢𝐨𝐧. When too many models share the same GPU cluster, the scheduler starts slicing compute and evicting caches.   This leads to context thrashing — where memory swaps cost more than inference.   At scale, this kills both performance and margins. So if you’re wondering why your inference bill doubled while latency stayed the same —   don’t blame the model.   Blame the infrastructure design. The real bottleneck isn’t model size — it’s architectural awareness.   Understanding schedulers, memory hierarchies, and caching strategies is what separates AI engineers from AI architects. And that’s exactly what we go deep into inside the Advanced System Design Cohort —   a 3-month, high-intensity program for Senior, Staff, and Principal Engineers who want to master the systems that power modern AI infra. You’ll learn to think beyond API calls — about how compute, caching, and scheduling interact to define scale and cost. If you’re ready to learn the architectures behind real AI systems —   there’s a form in the comments.   Apply, and we’ll check if you’re a great fit.   We’re selective, because this is where future technical leaders are being built.

I read a paper from NVIDIA Research last month that made a strong case for shifting from giant large language models (LLMs) to leaner, more specialized small language models (SLMs). I couldn’t agree more. https://lnkd.in/gbBNd_Bm Here are my top three takeaways: 1. Efficiency First – Models under 10B parameters consume fewer tokens, run faster, and cost significantly less to operate. Lower latency, reduced infrastructure demands, and greener AI. 2. Specialized Power – While large models excel at general conversation, small models shine in narrowly scoped tasks. Fine-tuning for a specific job can often match or exceed the performance of much larger models. 3. Better Fit for Agentic Systems – Most AI agents repeat structured, tool-based actions. SLMs are easier to fine-tune, deploy on-device, and integrate into modular multi-agent workflows, resulting in faster, cheaper, and more aligned systems. To test the theory, I built a specialized agent that generates a typical energy model based on building type and climate zone. I swapped between Qwen3:14B and Qwen3:4B on my local computer (M3, 18GB RAM). Running the same user query to generate results: Qwen3:14B – Input tokens: 3,052 | Output tokens: 2,070 | Duration: 164.24 s Qwen3:4B – Input tokens: 2,048 | Output tokens: 619 | Duration: 8.34 s That’s about 30% fewer tokens and 20× faster — achieving the same result. Sometimes, the future of AI is not about going bigger, but about going smaller, smarter, and faster. #AI #ArtificialIntelligence #MachineLearning #LLM #SLM #SmallLanguageModels #LargeLanguageModels #AgenticAI #MultiAgentSystems #EdgeAI #OnDeviceAI #NaturalLanguageProcessing #EnergyModeling #BuildingPerformance #EfficiencyInAI #TokenOptimization #ModelOptimization #AITesting #AIResearch

Stop comparing RAG and CAG. I wish I knew how each contributes to context before spending hours trying to get one do the job of other. Most teams are still trying to squeeze costs out of their RAG pipeline. But the smartest teams aren't just optimising, they're re-architecting their context. They know it’s not about RAG vs. CAG. It’s about knowing how to leverage each, intelligently. It's about Context Engineering. 𝗧𝗵𝗲 "𝗣𝗮𝘆-𝗣𝗲𝗿-𝗤𝘂𝗲𝗿𝘆" 𝗣𝗿𝗼𝗯𝗹𝗲𝗺: Retrieval-Augmented Generation (RAG) RAG is powerful, giving LLMs access to dynamic data. But from a cost perspective, it’s a “pay-per-drink” model. Every single query has a cost attached: • 𝗖𝗼𝗺𝗽𝘂𝘁𝗲 𝗖𝗼𝘀𝘁: API calls to an embedding model. • 𝗜𝗻𝗳𝗿𝗮𝘀𝘁𝗿𝘂𝗰𝘁𝘂𝗿𝗲 𝗖𝗼𝘀𝘁: Hosting a vector database and a retriever. • 𝗣𝗲𝗿𝗳𝗼𝗿𝗺𝗮𝗻𝗰𝗲 𝗖𝗼𝘀𝘁: Latency and irrelevant results degrade user experience, which costs you users.    Optimising RAG helps, but you're still paying for every single lookup. 𝗧𝗵𝗲 "𝗣𝗮𝘆-𝗢𝗻𝗰𝗲, 𝗨𝘀𝗲-𝗠𝗮𝗻𝘆" 𝗦𝗼𝗹𝘂𝘁𝗶𝗼𝗻: Cache-Augmented Generation (CAG) CAG flips the cost model on its head. It’s built for efficiency with scoped knowledge. Instead of fetching data every time, you: → Preload a static knowledge base into the model's context. → Compute and store its KV cache just once. → Reuse this cache across thousands of subsequent queries. The result is a massive drop in per-query costs. • 𝗕𝗹𝗮𝘇𝗶𝗻𝗴 𝗳𝗮𝘀𝘁: No real-time retrieval latency. • 𝗔𝗿𝗰𝗵𝗶𝘁𝗲𝗰𝘁𝘂𝗿𝗮𝗹𝗹𝘆 𝘀𝗶𝗺𝗽𝗹𝗲: Fewer moving parts to manage and pay for. • 𝗜𝗻𝗳𝗿𝗮-𝗹𝗶𝗴𝗵𝘁: The most expensive work (caching) is done upfront, not on every call. It’s Not RAG vs. CAG. It’s RAG + CAG. The most cost-effective AI systems don't choose one. They use a hybrid approach, like the teams at 𝗠𝗮𝗻𝘂𝘀 𝗔𝗜. The goal is to match the data's nature to the right architecture. This is 𝗖𝗼𝗻𝘁𝗲𝘅𝘁 𝗘𝗻𝗴𝗶𝗻𝗲𝗲𝗿𝗶𝗻𝗴: strategically deciding what knowledge is cached and what is retrieved. ✅ Use CAG for your static foundation: This is for knowledge that doesn't change often but is frequently accessed. Pay the upfront cost to cache it once and enjoy near-zero marginal cost for every query after. ✅ Use RAG for your dynamic layer: This is for information that is volatile, real-time, or user-specific. You only pay the retrieval cost when you absolutely need the freshest data. The Bottom Line Stop thinking in terms of "RAG vs. CAG." Start thinking like a 𝗖𝗼𝗻𝘁𝗲𝘅𝘁 𝗘𝗻𝗴𝗶𝗻𝗲𝗲𝗿. By building a static foundation with CAG and using RAG for dynamic lookups, you create a system that is not only powerful and fast but also dramatically more cost-effective at scale. RAG isn't dead, and CAG isn't a silver bullet. They are two essential tools in your cost-optimisation toolkit. If you're building an AI stack that's both smart and sustainable, this is for you. ♻️ Repost to share this strategy. ➕ Follow Shivani Virdi for more.

Analyzed a marketing team's AI usage last month. They burned through $3,400 in tokens. Only $890 worth actually produced usable content. That's 74% waste. Here's how to cut token waste without sacrificing output quality in 4 fixes: Fix 1: Stop Pasting Entire Documents Your team uploads 50-page PDFs for every prompt: ↳ AI reads all 50 pages every single time ↳ Only needs pages 12-14 for the actual task ↳ Wasting tokens on 47 irrelevant pages Smart approach: ↳ Extract only the relevant sections first ↳ Paste specific paragraphs instead of full docs ↳ Use document summaries for context One sales team cut token usage by 68% just by pasting meeting notes instead of full transcripts. Same output quality, fraction of the cost. Fix 2: Front-Load Your Context Most teams structure prompts backwards: ↳ Long explanation of what they want ↳ Then paste the context at the end ↳ AI processes everything before understanding the task Flip the structure: ↳ Context first (documents, data, examples) ↳ Instruction second (clear, specific task) ↳ Output format third (how you want it structured) Processing efficiency jumps 40%. Token waste drops proportionally. Fix 3: Use Projects and Memory Features Repeating the same context in every new chat: ↳ Brand guidelines pasted 47 times this month ↳ Product specs uploaded daily ↳ Company background in every prompt Claude Projects and ChatGPT memory solve this: ↳ Upload context once ↳ Reference it in every conversation ↳ Never pay to re-process the same information One client saved $1,200/month by moving their brand guide into a Project. Stop paying to teach AI the same things repeatedly. Fix 4: Refine Prompts Instead of Regenerating Team gets a bad output and hits regenerate: ↳ Same prompt, same context, same token cost ↳ Hoping for different results ↳ Burns through 3-5 generations before giving up Better approach: ↳ Analyze what's wrong with the output ↳ Adjust the prompt with specific fixes ↳ Regenerate once with improved instructions "Regenerate" is expensive trial and error. Refinement is strategic iteration. One content team reduced their average tokens per article from 8,400 to 3,100 just by improving prompts instead of spamming regenerate. Token costs are controllable. Most waste comes from inefficient workflows, not necessary usage. Audit your team's AI habits this week. The waste will fund better tools next quarter. What's your biggest source of token waste? P.S. Want to learn more about AI? 1. Scroll to the top 2. Click "Visit my website" 3. Sign-up for our free newsletter

AI Cost Optimization: 27% Growth Demands Planning The concept of Lean AI is another essential perspective in cost optimization. Lean AI focuses on developing smaller, more efficient AI models tailored to a company’s specific operational needs. These models require less data and computational power to train and run, markedly reducing costs compared to large, generalized AI models. By solving specific problems with precisely tailored solutions, enterprises can avoid the unnecessary expenditure associated with overcomplicated AI systems. Starting with these smaller, targeted applications allows organizations to incrementally build on their AI capabilities and ensure that each step is cost-justifiable and closely tied to its potential value. Companies can progressively expand AI capabilities through a Lean AI approach, making cost management a central consideration. Efficiently optimizing computational resources plays another critical role in controlling AI expenses. Monitor and manage computing resources to ensure the company only pays for what it needs. Tools that track compute usage can highlight inefficiencies and help make more informed decisions about scaling resources.

I had the opportunity to teach a Pavilion cohort with Yash Tekriwal 🤔 and here are 11 outbound tips we went over when automating your workflows! I hit the character limit on this post so if you have any questions about the nuance, don't hesitate to comment as there's for sure a lot here. 1. "Personalization" should = 10 minutes of what you would manually research The most common mistake is asking “What are other companies doing with AI?” Instead, ask “What should we be doing with AI?” Think about how you’d research a company for 10 minutes: what you’d check, how those checks change your message (hook, value, CTA), and then how to automate that same research. Rule of thumb: Claygent can usually handle most of the research you need. 2. Generate AI snippets, not AI emails Long, “all-in-one” prompts rarely nail everything. It’s much easier to create strong outputs in small pieces (e.g., 2–7 words or one short sentence at a time) and assemble the final message. 3. Don’t ever tell the model you’re writing a “cold email” Describe the persona and tone you want instead. This avoids the generic “sales-blog” style many models default to. 4. One job per prompt Every AI in your workflow should do one task at a time. Claygent does one research task. Your writing prompt does one writing task. Combining tasks creates errors and cost creep. 5. Feed actual data into the prompt Don’t say “This is salesforce.com, write an email.” Give the model the specific facts you’d use yourself. The more grounded the inputs, the less guessing. 6. Examples are the most important input for your prompts. Great examples can turn an okay prompt—and an okay system prompt—into something excellent. Curate 2–3 gold-standard input→output examples and keep them fresh. Remove weak ones. Examples anchor style, brevity, and relevance. 7. Always use conditional formulas/run-conditions to protect spend Only run research when required inputs exist; only run writing when research fields are filled. Guardrails reduce wasted credits and noisy outputs. 8. Dictate your prompts If typing slows you down, speak your prompt, then ask the model to tighten and clarify it. You’ll capture more nuance, faster. 9. Ask AI to critique your prompt Paste your draft prompt + goal and ask: “Do you have any follow-up questions to help make this more accurate?” Answer them, then have the model rewrite the prompt with fewer tokens. 10. Model + cost hygiene We use GPT-4o mini for most tasks, and we lock our API key to that model so the team can’t accidentally run expensive ones. Keep prompts short and examples essential to control token spend. 11. Ban filler/jargon with a general system prompt Keep a fifth-grade reading level, short sentences, and concrete verbs. Ask ChatGPT to draft a reusable system prompt that bans words like “leverage,” “synergy,” “innovate,” etc., and apply that system prompt to any AI that writes email snippets.

Over the next three years, the U.S. will need the equivalent of three New York City's worth of energy to support AI. As someone who cares about climate change, how can you reduce the environmental impact of your LLM use? Energy use by LLMs is impacted by model Size (number of parameters), the length of your input and the model’s output (input and output tokens), and model optimizations like pruning seen in newer models. Here’s how to reduce energy usage: - For text summarization or quick Information Retrieval, smaller models like GPT-4o Mini or GPT-o1 Mini are sufficient. These tasks don't require the full power of larger models. - Creative Writing or Complex Analysis: For tasks requiring nuance, opt for GPT-4o. However, consider whether splitting the task into smaller, simpler components might allow you to use a smaller model.  - Testing and Experimentation: If you’re experimenting, start with a smaller model (GPT-4o Mini or GPT-o1 Mini). Upgrade if the results are insufficient. - For developers accessing models through the API, smaller models are not only more energy-efficient but also more cost-effective. Start with those (or break up your task into components and mix and match models of different sizes). More details on LLM use for the climate conscious here: https://lnkd.in/eT-HHk8T