Optimizing LLM Inference for Maximum Efficiency

Most popular LLMs—such as the GPT series from OpenAI and LLaMA from Meta—are designed using a decoder-only architecture, which predicts the next token based on the previous N tokens. This architecture is primarily intended for text generation tasks.

During inference, three main steps are involved:

1. Embedding & Positional Encoding The input prompt is transformed into embeddings using the embedding layer, followed by encoding positional information through the positional encoding layer (modern LLMs often use RoPE – rotary positional encoding).
2. Contextual Enhancement The encoded text is further refined using multi-head attention.
3. Sequential Token Generation Output tokens are generated one at a time in sequence.

The first two steps are computationally expensive, involving matrix multiplications optimized for hardware like GPUs and TPUs. However, step 3 introduces a bottleneck: token generation is inherently sequential because each token depends on previously generated tokens. This limits parallelization and underutilizes available hardware.

When deploying fine-tuned LLM-based applications, your goal is to deliver accurate content with the lowest latency while maximizing hardware utilization. Below are key techniques to optimize inference:

1. KV Cache

Each new token depends on prior context. For example, generating the 100th token requires tokens 1–99; the 101st token requires tokens 1–100. This repeated computation is inefficient. Solution: Cache the Key-Value (KV) vectors produced by the self-attention layer for each token. Instead of recomputing, retrieve cached vectors and append new ones. Challenge: KV cache size is dynamic and grows with token count, leading to memory constraints such as memory fragmentation and allocation overhead—especially for large models. To address this, use static KV caching by pre-allocating memory based on maximum context length, embedding size, and number of layers.

2. Continuous Batching

Batching multiple requests improves throughput by transferring more data to GPUs at once. Problem: Decoder-only models face variability in input/output lengths, causing idle hardware when shorter requests finish early. Solution: Use continuous batching (in-flight mode), where completed requests are replaced with new ones, keeping accelerators fully utilized. Periodic pauses may still occur for embedding/encoding of incoming requests.

3. Speculative Decoding

To accelerate token generation and saturate parallel processing, leverage speculative decoding (assistant generation). Approach: A smaller model predicts multiple tokens in parallel. These speculative completions are validated by the original model in batches. The longest matching sequence is retained; incorrect predictions are discarded.

4. Paged Attention

Transformer models scale quadratically with token count, increasing KV cache size requirements. Solution: Paged Attention partitions KV cache into smaller blocks, similar to OS virtual memory paging. Blocks are fetched efficiently during attention computation, improving memory utilization and enabling larger batches. Paged Attention also supports memory sharing across outputs from the same prompt—especially useful for beam search—reducing redundancy and memory overhead.

Bottom Line: By implementing KV caching, continuous batching, speculative decoding, and paged attention, you can significantly optimize LLM inference, reduce latency, and maximize hardware efficiency.

This article is cited from the LLM Engineer's Handbook by Paul Lusztin and Maxime Labonne.