How to Optimize Machine Learning Performance

If you’re an AI engineer trying to optimize your LLMs for inference, here’s a quick guide for you 👇 Efficient inference isn’t just about faster hardware, it’s a multi-layered design problem. From how you compress prompts to how your memory is managed across GPUs, everything impacts latency, throughput, and cost. Here’s a structured taxonomy of inference-time optimizations for LLMs: 1. Data-Level Optimization Reduce redundant tokens and unnecessary output computation. → Input Compression:  - Prompt Pruning, remove irrelevant history or system tokens  - Prompt Summarization, use model-generated summaries as input  - Soft Prompt Compression, encode static context using embeddings  - RAG, replace long prompts with retrieved documents plus compact queries → Output Organization:  - Pre-structure output to reduce decoding time and minimize sampling steps 2. Model-Level Optimization (a) Efficient Structure Design → Efficient FFN Design, use gated or sparsely-activated FFNs (e.g., SwiGLU) → Efficient Attention, FlashAttention, linear attention, or sliding window for long context → Transformer Alternates, e.g., Mamba, Reformer for memory-efficient decoding → Multi/Group-Query Attention, share keys/values across heads to reduce KV cache size → Low-Complexity Attention, replace full softmax with approximations (e.g., Linformer) (b) Model Compression → Quantization:  - Post-Training, no retraining needed  - Quantization-Aware Training, better accuracy, especially <8-bit → Sparsification:  - Weight Pruning, Sparse Attention → Structure Optimization:  - Neural Architecture Search, Structure Factorization → Knowledge Distillation:  - White-box, student learns internal states  - Black-box, student mimics output logits → Dynamic Inference, adaptive early exits or skipping blocks based on input complexity 3. System-Level Optimization (a) Inference Engine → Graph & Operator Optimization, use ONNX, TensorRT, BetterTransformer for op fusion → Speculative Decoding, use a smaller model to draft tokens, validate with full model → Memory Management, KV cache reuse, paging strategies (e.g., PagedAttention in vLLM) (b) Serving System → Batching, group requests with similar lengths for throughput gains → Scheduling, token-level preemption (e.g., TGI, vLLM schedulers) → Distributed Systems, use tensor, pipeline, or model parallelism to scale across GPUs My Two Cents 🫰 → Always benchmark end-to-end latency, not just token decode speed → For production, 8-bit or 4-bit quantized models with MQA and PagedAttention give the best price/performance → If using long context (>64k), consider sliding attention plus RAG, not full dense memory → Use speculative decoding and batching for chat applications with high concurrency → LLM inference is a systems problem. Optimizing it requires thinking holistically, from tokens to tensors to threads. Image inspo: A Survey on Efficient Inference for Large Language Models ---- Follow me (Aishwarya Srinivasan) for more AI insights!

Recently helped a client cut their AI development time by 40%. Here’s the exact process we followed to streamline their workflows. Step 1: Optimized model selection using a Pareto Frontier. We built a custom Pareto Frontier to balance accuracy and compute costs across multiple models. This allowed us to select models that were not only accurate but also computationally efficient, reducing training times by 25%. Step 2: Implemented data versioning with DVC. By introducing Data Version Control (DVC), we ensured consistent data pipelines and reproducibility. This eliminated data drift issues, enabling faster iteration and minimizing rollback times during model tuning. Step 3: Deployed a microservices architecture with Kubernetes. We containerized AI services and deployed them using Kubernetes, enabling auto-scaling and fault tolerance. This architecture allowed for parallel processing of tasks, significantly reducing the time spent on inference workloads. The result? A 40% reduction in development time, along with a 30% increase in overall model performance. Why does this matter? Because in AI, every second counts. Streamlining workflows isn’t just about speed—it’s about delivering superior results faster. If your AI projects are hitting bottlenecks, ask yourself: Are you leveraging the right tools and architectures to optimize both speed and performance?

🐢🚀 Making GPUs Go Brrr: The Art of Deep Learning Optimization TL;DR 🧠 Deep learning performance depends on three bottlenecks: compute, memory bandwidth, and overhead. Optimizing requires identifying which regime you're in. 🏭 Compute-bound: Maximize Tensor Core usage (e.g., matmuls) to achieve up to 312 TFLOPS.  🚚 Memory-bound: Use operator fusion to reduce costly memory transfers (e.g., x.cos().cos() is 2x faster when fused). 🐢 Overhead-bound: Framework and Python dispatch costs dominate small ops. Use tracing (jit.trace) or TorchDynamo to reduce overhead. Problems and Solutions 🐢 Overhead-bound: Use TorchDynamo or CUDA Graphs to reduce Python and framework dispatch costs. 🚚 Memory-bound: Fuse operations (e.g., NVFuser) to avoid repeated memory reads/writes. 🏭 Compute-bound: Focus on Tensor Core utilization for matrix multiplications, as non-matmul operations are 15x slower. Experiments & Setup ⏱️ PyTorch profiler: Reveals GPU idle gaps caused by CPU overhead (pink CPU vs. green GPU traces). 📦 Batch size test: Doubling batch size with only a 10% runtime increase indicates overhead-bound operations. 🧮 FLOP counting: Non-matmul ops (e.g., layer norm) consume 0.2% of FLOPs but achieve 250x less efficiency. Novel Insights 🧩 Operator fusion: Fused gelu costs are similar to relu due to reduced memory transfers. 🔄 Rematerialization: Recomputation can reduce both memory and runtime, as seen in AOTAutograd's min-cut optimization. 📉 Hardware disparity: GPU compute grows faster than memory bandwidth, making memory optimizations increasingly critical. Improvements Over Prior Work 🧪 TorchDynamo: A JIT compiler that dynamically reduces Python overhead without sacrificing flexibility. 🚀 CUDA Graphs: Eliminates kernel launch overhead but requires static execution. [Source: Chunk 10] 🔧 NVFuser: Automates operator fusion for pointwise/reduction ops, achieving 2x speedups in some cases. Key Architecture Details 🧠 Tensor Cores: Specialized for matmuls, achieving 312 TFLOPS, compared to 19.5 TFLOPS for general CUDA cores. 📦 Memory hierarchy: DRAM (global) → SRAM (shared) → registers. Operator fusion minimizes DRAM usage.  🔄 Asynchronous execution: CPU queues GPU kernels to hide overhead, but small ops leave GPUs idle. Future Work 🤖 JIT compilers: Combine flexibility and low overhead with VM-level introspection (e.g., TorchDynamo). 🧩 Hardware-software co-design: Optimize for non-matmul ops, especially on TPUs. 📉 Memory-aware training: Automate rematerialization using min-cut algorithms. Key Visualizations 🏭 Factory analogy: Compute = factory, memory = warehouse, bandwidth = shipping. Optimizing compute means reducing shipping delays. 🔥 Flamegraph: Shows that 90% of PyTorch a + b time is overhead, not actual computation. 📈 Microbenchmark plot: Increasing compute intensity (e.g., repeat=64) shifts operations from memory-bound (0.2 TFLOPS) to compute-bound (9.75 TFLOPS). 👇

When working with 𝗟𝗟𝗠𝘀, most discussions revolve around improving 𝗺𝗼𝗱𝗲𝗹 𝗮𝗰𝗰𝘂𝗿𝗮𝗰𝘆, but there’s another equally critical challenge: 𝗹𝗮𝘁𝗲𝗻𝗰𝘆. Unlike traditional systems, these models require careful orchestration of multiple stages, from processing prompts to delivering output, each with its own unique bottlenecks. Here’s a 5-step process to minimize latency effectively:  1️⃣ 𝗣𝗿𝗼𝗺𝗽𝘁 𝗣𝗿𝗼𝗰𝗲𝘀𝘀𝗶𝗻𝗴: Optimize by caching repetitive prompts and running auxiliary tasks (e.g., safety checks) in parallel.  2️⃣ 𝗖𝗼𝗻𝘁𝗲𝘅𝘁 𝗣𝗿𝗼𝗰𝗲𝘀𝘀𝗶𝗻𝗴: Summarize and cache context, especially in multimodal systems. 𝘌𝘹𝘢𝘮𝘱𝘭𝘦: 𝘐𝘯 𝘥𝘰𝘤𝘶𝘮𝘦𝘯𝘵 𝘴𝘶𝘮𝘮𝘢𝘳𝘪𝘻𝘦𝘳𝘴, 𝘤𝘢𝘤𝘩𝘪𝘯𝘨 𝘦𝘹𝘵𝘳𝘢𝘤𝘵𝘦𝘥 𝘵𝘦𝘹𝘵 𝘦𝘮𝘣𝘦𝘥𝘥𝘪𝘯𝘨𝘴 𝘴𝘪𝘨𝘯𝘪𝘧𝘪𝘤𝘢𝘯𝘵𝘭𝘺 𝘳𝘦𝘥𝘶𝘤𝘦𝘴 𝘭𝘢𝘵𝘦𝘯𝘤𝘺 𝘥𝘶𝘳𝘪𝘯𝘨 𝘪𝘯𝘧𝘦𝘳𝘦𝘯𝘤𝘦.  3️⃣ 𝗠𝗼𝗱𝗲𝗹 𝗥𝗲𝗮𝗱𝗶𝗻𝗲𝘀𝘀: Avoid cold-boot delays by preloading models or periodically waking them up in resource-constrained environments.  4️⃣ 𝗠𝗼𝗱𝗲𝗹 𝗣𝗿𝗼𝗰𝗲𝘀𝘀𝗶𝗻𝗴: Focus on metrics like 𝗧𝗶𝗺𝗲 𝘁𝗼 𝗙𝗶𝗿𝘀𝘁 𝗧𝗼𝗸𝗲𝗻 (𝗧𝗧𝗙𝗧) and 𝗜𝗻𝘁𝗲𝗿-𝗧𝗼𝗸𝗲𝗻 𝗟𝗮𝘁𝗲𝗻𝗰𝘆 (𝗜𝗧𝗟). Techniques like 𝘁𝗼𝗸𝗲𝗻 𝘀𝘁𝗿𝗲𝗮𝗺𝗶𝗻𝗴 and 𝗾𝘂𝗮𝗻𝘁𝗶𝘇𝗮𝘁𝗶𝗼𝗻 can make a big difference.  5️⃣ 𝗢𝘂𝘁𝗽𝘂𝘁 𝗔𝗻𝗮𝗹𝘆𝘀𝗶𝘀: Stream responses in real-time and optimize guardrails to improve speed without sacrificing quality. It’s ideal to think about latency optimization upfront, avoiding the burden of tech debt or scrambling through 'code yellow' fire drills closer to launch. Addressing it systematically can significantly elevate the performance and usability of LLM-powered applications. #AI #LLM #MachineLearning #Latency #GenerativeAI

Let's cut through the hype. If you're building AI-powered search, you've probably heard that bigger embedding models are always better. That's not the full story. Here's what I've learned from real-world implementations: Lightweight embeddings + reranking often outperform massive embedding models alone. This combo can dramatically reduce latency and infrastructure costs, especially at scale. Vector quantization is your friend. It allows you to handle larger datasets without proportionally increasing compute requirements. The key insight: Reranking allows you to be smarter about where you allocate computational resources. Instead of using a huge model to embed everything, you can: Use a smaller, faster model for initial retrieval Apply a more sophisticated reranker only to the top results Quantize vectors to optimize storage and retrieval This approach scales better and often yields better results. Why? Because rerankers can capture nuanced query-document relationships that even large embedding models might miss. Practical tips: Evaluate rerankers on your specific data. Benchmark scores can be misleading. Watch reranking latency. It can add 50-500ms per query if you are using an external provider like Cohere or VoyageAI. A library like text-embedding-inference can allow you to rerank in under 10ms: https://lnkd.in/gzt37Y3M Consider fine-tuning rerankers on domain-specific data. 20-40% performance gains aren't uncommon. Fine-tuning a reranker might give you better results than fine-tuning an embedding model, although both strategies perform well and can be used in tandem. Remember: In production, a "good enough" embedding model with smart reranking often beats a state-of-the-art embedding model used naively.

If you want to really improve performance in sentiment analysis or sequence classification, go data-centric: not model-centric. Hold your temptation on quick and dirty LLM prompt hacks for data augmentation. After I've mentored hundreds of NLP students at Stanford Online AI program, I’ve seen a recent recurring pattern: projects plateauing in performance not because of model choice, but because of the data strategy. One of the most effective upgrades? Robust data augmentation tailored to your domain. When applied right, it can unlock 10+ point gains in macro F1 score - without even changing your model. Here are 3 techniques that consistently outperform generic prompt-based generation: 1. Back translation for natural variation Round-trip translation (e.g., English → Portuguese → Serbian → back) creates diverse, domain-consistent examples - enhancing your training data without losing label alignment and consistency. 2. Contextual augmentation via Masked Language Models (MLM) Use BERT or RoBERTa to inject controlled perturbations. These substitutions maintain syntax and sentiment while expanding your training distribution. 3. Finetuning with class weights For imbalanced datasets, weighting underrepresented classes during training can significantly improve classification metrics, especially in multi-class sentiment tasks. The main takeaway? LLM prompting may seem appealing. But data-centric approaches still matter more and deliver better results - especially when working with domain-specific sentiment or classification pipelines. Put your effort for your AI systems where the impact is: your data.

In 2019, we were working on a machine learning project to predict the optimal size of a pick list for a picker in a warehouse. Typical Supplychain101 Problem. We spent months tuning the model—trying different algorithms, adjusting hyperparameters, and experimenting nonstop. But no matter what we tried, the model’s performance barely improved. Frustrated, we finally stepped back and looked at the data. It turned out we weren’t collecting some of the most important features that directly influenced the picker’s performance—factors like real-time workload, aisle congestion, and picker experience. No amount of model tuning could overcome the fact that critical information was missing. What if I tell you 🤔 that in machine learning, better data 📊 often beats better algorithms ⚙️—but we rarely talk about it 💬?? Today, intelligent algorithms get all the attention—larger models, more compute, smarter architectures. But as Chip Huyen points out in “Designing Machine Learning Systems,” investing in better data often has a much bigger impact than trying to squeeze out a few extra points of accuracy from a fancier model. The reality is: There's a clear, structured way to improve algorithms—through research papers, benchmarks, and tuning strategies. But improving data —> adding missing features —> embedding domain expertise — requires deep business understanding and manual effort. It’s harder, messier, and doesn’t make headlines. Most organizations don’t have unlimited data or deep domain knowledge. And that’s why so many ML systems hit performance ceilings despite model upgrades. The ML systems that truly succeed at scale invest in: ✅ Thoughtful feature engineering ✅ Continuous data enrichment ✅ Rigorous validation ✅ Human-in-the-loop feedback systems Next time you’re trying to boost your model’s performance, ask: “Is it the model—or is it the data which has more predictive power is missing from your feature space?” Because even the most intelligent algorithm can only learn from the information you provide. Curious to hear your experiences: When did better data (not a better model) make all the difference for you? What strategies do you use to make data better? #MachineLearning #DataScience #MLSystemDesign #AI #DataQuality

Your LLM isn't slow because of the model. It's slow because you skipped the optimization layer Here are 16 techniques to make LLMs faster — ranked by where to start. Start here (highest ROI): → Quantization — biggest speed gain, easiest to implement → Flash Attention — should be default by now → KV-Cache Quantization — massive memory savings → Batching & Dynamic Batching — free speed if you're serving Then here: → Mixed Precision Inference — float16 where it won't break → Speculative Decoding — 2-3x speedup when it works → Paged Attention — vLLM uses this for a reason Then here: → Tensor Parallelism — when one GPU isn't enough → Pipeline Parallelism — when you have multiple GPUs → Model Serving Optimization — TensorRT, ONNX, the infra layer Advanced (when basics are done): → Pruning — requires retraining → Knowledge Distillation — requires a teacher model → LoRA — for fine-tuning, not inference speed → Weight Sharing — architecture-level change → Sparse Attention — model-specific → Early Exit — experimental but promising Most people start at the bottom. Then wonder why nothing changed. Start at the top. Work down. ♻️ Repost to save someone from optimizing the wrong thing first.

Embeddings eat up storage. Processing slows down. Search gets expensive. Here's how to optimize without breaking things: → Model Selection Pick the right size for your use case. Smaller models work for simple tasks. Larger ones handle complex semantic search. → Dimensionality Reduction Cut dimensions without losing meaning. 768 → 384 dimensions saves 50% storage. Test accuracy before committing. → Quantization Convert float32 to int8. 4x storage reduction. Minimal accuracy loss. → Batch Processing Process embeddings in groups. Faster than one-by-one. Better GPU utilization. → Caching Strategy Store frequently used embeddings. Skip redundant computations. Speed up retrieval by 10x. → Update vs Rebuild Incremental updates for small changes. Full rebuild when data shifts significantly. Track drift to decide. → Multi-lingual Handling Use cross-lingual models for global data. Separate embeddings per language if needed. Balance cost and accuracy. The difference between slow systems and fast ones? Optimization decisions made early. 🔄 Repost this if embeddings optimization has been on your radar. ➡️ Follow Aditya for insights on AI engineering that cut through the complexity.

Day 8/30 of LLMs/SLMs - Training LLMs at Scale Training LLMs isn’t just about brute force compute — it’s about engineering tricks that make the impossible possible. When you hear that models like GPT-4 or LLaMA were trained on hundreds of billions of tokens, it’s easy to imagine endless racks of GPUs chewing through data. But the truth is: without optimization techniques, even the largest clusters would run out of memory or grind to a halt. Four of the most important techniques are 𝐠𝐫𝐚𝐝𝐢𝐞𝐧𝐭 𝐜𝐡𝐞𝐜𝐤𝐩𝐨𝐢𝐧𝐭𝐢𝐧𝐠, 𝐦𝐢𝐱𝐞𝐝 𝐩𝐫𝐞𝐜𝐢𝐬𝐢𝐨𝐧, 𝐙𝐞𝐑𝐎, 𝐚𝐧𝐝 𝐅𝐥𝐚𝐬𝐡𝐀𝐭𝐭𝐞𝐧𝐭𝐢𝐨𝐧. Gradient checkpointing trades memory for compute. Normally, backpropagation requires storing all intermediate activations. With checkpointing, you only store a subset, and recompute the rest on the backward pass. Docs - https://lnkd.in/gX4ub4pw Mixed precision training uses half-precision (FP16 or BF16) instead of full FP32 for most operations. The payoff is twofold: faster computation (modern GPUs are optimized for it) and reduced memory usage. For example, switching to BF16 can nearly double throughput while keeping numerical stability intact. Docs - https://lnkd.in/gftvMXRk ZeRO (Zero Redundancy Optimizer) from DeepSpeed takes a more radical approach: instead of replicating model states across all GPUs, it shards them. Gradients, optimizer states, and parameters are split across devices. This means no single GPU needs to hold the entire model. Tutorial - https://lnkd.in/gjave7V6 FlashAttention is a recent innovation that rethinks how attention is computed. The classic implementation wastes memory by materializing giant intermediate matrices. FlashAttention computes attention in chunks directly in GPU SRAM, reducing memory usage and increasing speed. On real-world workloads, this can mean 2–4× faster training without changing model quality. Docs - https://lnkd.in/gBQBzzgE Together, these techniques are why we can train models with billions of parameters on clusters of thousands of GPUs without blowing past memory and compute limits. Tune in tomorrow for more SLM/LLMs deep dives. -- 🚶➡️ To learn more about LLMs/SLMs, follow me - Karun! ♻️ Share so others can learn, and you can build your LinkedIn presence! (Img Src: https://lnkd.in/gCNwAZT7)