Optimizing LLM Output Using APO Techniques

🚀 Excited to share our latest research on accelerated generation techniques for large language models (LLMs)! 🧠✨ 🔗 https://lnkd.in/gRPd2MaV In our comprehensive survey, we delve into 30+ techniques to speed up text generation, making real-time applications more efficient. Accelerated generation techniques aim to reduce the time and computational resources needed for LLMs to generate text, ensuring faster and more responsive AI systems. Here's a sneak peek: - Speculative Decoding: Generates multiple candidate outputs simultaneously to reduce latency. For example, SpecDec achieves up to a 5x speedup in generation. - Early Exiting Mechanisms: Terminates the generation process upon confident predictions, saving computational resources. CALM dynamically allocates resources per input, cutting down processing time. - Non-Autoregressive Methods: Innovates parallelization for faster, coherent output generation. FlowSeq leverages latent variables to model dependencies while maintaining efficiency. This paper, created in collaboration with researchers from Massachusetts Institute of Technology and Columbia University, is crucial for advancing LLM efficiency and enhancing their real-world applications. Dive into the full details and explore the cutting-edge techniques driving the future of AI! ✍🏻 Mahsa Khoshnoodi, Vinija Jain, Mingye Gao, Malavika Srikanth, and Aman Chadha

Last week, I described four design patterns for AI agentic workflows that I believe will drive significant progress: Reflection, Tool use, Planning and Multi-agent collaboration. Instead of having an LLM generate its final output directly, an agentic workflow prompts the LLM multiple times, giving it opportunities to build step by step to higher-quality output. Here, I'd like to discuss Reflection. It's relatively quick to implement, and I've seen it lead to surprising performance gains. You may have had the experience of prompting ChatGPT/Claude/Gemini, receiving unsatisfactory output, delivering critical feedback to help the LLM improve its response, and then getting a better response. What if you automate the step of delivering critical feedback, so the model automatically criticizes its own output and improves its response? This is the crux of Reflection. Take the task of asking an LLM to write code. We can prompt it to generate the desired code directly to carry out some task X. Then, we can prompt it to reflect on its own output, perhaps as follows: Here’s code intended for task X: [previously generated code] Check the code carefully for correctness, style, and efficiency, and give constructive criticism for how to improve it. Sometimes this causes the LLM to spot problems and come up with constructive suggestions. Next, we can prompt the LLM with context including (i) the previously generated code and (ii) the constructive feedback, and ask it to use the feedback to rewrite the code. This can lead to a better response. Repeating the criticism/rewrite process might yield further improvements. This self-reflection process allows the LLM to spot gaps and improve its output on a variety of tasks including producing code, writing text, and answering questions. And we can go beyond self-reflection by giving the LLM tools that help evaluate its output; for example, running its code through a few unit tests to check whether it generates correct results on test cases or searching the web to double-check text output. Then it can reflect on any errors it found and come up with ideas for improvement. Further, we can implement Reflection using a multi-agent framework. I've found it convenient to create two agents, one prompted to generate good outputs and the other prompted to give constructive criticism of the first agent's output. The resulting discussion between the two agents leads to improved responses. Reflection is a relatively basic type of agentic workflow, but I've been delighted by how much it improved my applications’ results. If you’re interested in learning more about reflection, I recommend: - Self-Refine: Iterative Refinement with Self-Feedback, by Madaan et al. (2023) - Reflexion: Language Agents with Verbal Reinforcement Learning, by Shinn et al. (2023) - CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing, by Gou et al. (2024) [Original text: https://lnkd.in/g4bTuWtU ]

If you’re an AI engineer trying to optimize your LLMs for inference, here’s a quick guide for you 👇 Efficient inference isn’t just about faster hardware, it’s a multi-layered design problem. From how you compress prompts to how your memory is managed across GPUs, everything impacts latency, throughput, and cost. Here’s a structured taxonomy of inference-time optimizations for LLMs: 1. Data-Level Optimization Reduce redundant tokens and unnecessary output computation. → Input Compression:  - Prompt Pruning, remove irrelevant history or system tokens  - Prompt Summarization, use model-generated summaries as input  - Soft Prompt Compression, encode static context using embeddings  - RAG, replace long prompts with retrieved documents plus compact queries → Output Organization:  - Pre-structure output to reduce decoding time and minimize sampling steps 2. Model-Level Optimization (a) Efficient Structure Design → Efficient FFN Design, use gated or sparsely-activated FFNs (e.g., SwiGLU) → Efficient Attention, FlashAttention, linear attention, or sliding window for long context → Transformer Alternates, e.g., Mamba, Reformer for memory-efficient decoding → Multi/Group-Query Attention, share keys/values across heads to reduce KV cache size → Low-Complexity Attention, replace full softmax with approximations (e.g., Linformer) (b) Model Compression → Quantization:  - Post-Training, no retraining needed  - Quantization-Aware Training, better accuracy, especially <8-bit → Sparsification:  - Weight Pruning, Sparse Attention → Structure Optimization:  - Neural Architecture Search, Structure Factorization → Knowledge Distillation:  - White-box, student learns internal states  - Black-box, student mimics output logits → Dynamic Inference, adaptive early exits or skipping blocks based on input complexity 3. System-Level Optimization (a) Inference Engine → Graph & Operator Optimization, use ONNX, TensorRT, BetterTransformer for op fusion → Speculative Decoding, use a smaller model to draft tokens, validate with full model → Memory Management, KV cache reuse, paging strategies (e.g., PagedAttention in vLLM) (b) Serving System → Batching, group requests with similar lengths for throughput gains → Scheduling, token-level preemption (e.g., TGI, vLLM schedulers) → Distributed Systems, use tensor, pipeline, or model parallelism to scale across GPUs My Two Cents 🫰 → Always benchmark end-to-end latency, not just token decode speed → For production, 8-bit or 4-bit quantized models with MQA and PagedAttention give the best price/performance → If using long context (>64k), consider sliding attention plus RAG, not full dense memory → Use speculative decoding and batching for chat applications with high concurrency → LLM inference is a systems problem. Optimizing it requires thinking holistically, from tokens to tensors to threads. Image inspo: A Survey on Efficient Inference for Large Language Models ---- Follow me (Aishwarya Srinivasan) for more AI insights!

Fascinating new research paper on Large Language Model Acceleration through KV Cache Management! A comprehensive survey has emerged from researchers at The Hong Kong Polytechnic University, The Hong Kong University of Science and Technology, and other institutions, diving deep into how we can make LLMs faster and more efficient through Key-Value cache optimization. The paper breaks down KV cache management into three critical levels: >> Token-Level Innovations - Static and dynamic cache selection strategies - Intelligent budget allocation across model layers - Advanced cache merging techniques - Mixed-precision quantization approaches - Low-rank matrix decomposition methods >> Model-Level Breakthroughs - Novel attention grouping and sharing mechanisms - Architectural modifications for better cache utilization - Integration of non-transformer architectures >> System-Level Optimizations - Sophisticated memory management techniques - Advanced scheduling algorithms - Hardware-aware acceleration strategies What's particularly interesting is how the researchers tackle the challenges of long-context processing. They present innovative solutions like dynamic token selection, mixed-precision quantization, and cross-layer cache sharing that can dramatically reduce memory usage while maintaining model performance. The paper also explores cutting-edge techniques like attention sink mechanisms, beehive-like structures for cache management, and adaptive hybrid compression strategies that are pushing the boundaries of what's possible with LLM inference. A must-read for anyone working in AI optimization, model acceleration, or large-scale language model deployment. The comprehensive analysis and taxonomies provided make this an invaluable resource for both researchers and practitioners in the field.

Are your LLM apps still hallucinating? Zep used to as well—a lot. Here’s how we worked to solve Zep's hallucinations. We've spent a lot of cycles diving into why LLMs hallucinate and experimenting with the most effective techniques to prevent it. Some might sound familiar, but it's the combined approach that really moves the needle. First, why do hallucinations happen? A few core reasons: 🔍 LLMs rely on statistical patterns, not true understanding. 🎲 Responses are based on probabilities, not verified facts. 🤔 No innate ability to differentiate truth from plausible fiction. 📚 Training datasets often include biases, outdated info, or errors. Put simply: LLMs predict the next likely word—they don’t actually "understand" or verify what's accurate. When prompted beyond their knowledge, they creatively fill gaps with plausible (but incorrect) info. ⚠️ Funny if you’re casually chatting—problematic if you're building enterprise apps. So, how do you reduce hallucinations effectively? The #1 technique: grounding the LLM in data. - Use Retrieval-Augmented Generation (RAG) to anchor responses in verified data. - Use long-term memory systems like Zep to ensure the model is always grounded in personalization data: user context, preferences, traits etc - Fine-tune models on domain-specific datasets to improve response consistency and style, although fine-tuning alone typically doesn't add substantial new factual knowledge. - Explicit, clear prompting—avoid ambiguity or unnecessary complexity. - Encourage models to self-verify conclusions when accuracy is essential. - Structure complex tasks with chain-of-thought prompting (COT) to improve outputs or force "none"/unknown responses when necessary. - Strategically tweak model parameters (e.g., temperature, top-p) to limit overly creative outputs. - Post-processing verification for mission-critical outputs, for example, matching to known business states. One technique alone rarely solves hallucinations. For maximum ROI, we've found combining RAG with a robust long-term memory solution (like ours at Zep) is the sweet spot. Systems that ground responses in factual, evolving knowledge significantly outperform. Did I miss any good techniques? What are you doing in your apps?

When optimizing prompt for #LLM, what if we don't have a score to evaluate performance of every prompt? The work of Xiaoqiang Lin, Zhongxiang Dai, Arun Verma, See-Kiong Ng, Patrick Jaillet, and Bryan Kian Hsiang Low achieves prompt optimization with human feedback (APOHF) only. APOHF uses a dueling bandit strategy to choose a pair of prompts in every iteration to generate their responses which are then shown to the user for preference feedback. In practice, the user only needs to give an initial task description & then a series of preference feedback. We use APOHF to optimize the prompt for #DALLE3 while only asking user for preference feedback between pairs of generated images. APOHF can efficiently produce an image which aligns well with the user's preference (compare ground truth image and image at iteration 10) while only requiring a small number of human feedback instances. APOHF can be adapted to solve response optimization with human feedback. For every received prompt, let the LLM generate many responses. Then, adapt APOHF to choose a pair of responses to query the user for preference feedback. As shown in the table, the response discovered by APOHF aligns well with human preferences as it is well organized (via a numbered list) and detailed. https://lnkd.in/g_Gv5Wse #LLMs #RLHF

Prompt engineering requires a lot of manual effort. Here are four automatic prompt optimization algorithms that can help to improve your prompt with minimal effort… (1) Automatic Prompt Engineer (APE) [1] searches over a pool of prompts proposed by an LLM–usually ~32-64 prompts–to find the prompt that performs best. This setup uses separate LLMs to propose and evaluate prompts. For evaluation, we generate output via zero-shot inference and evaluate the output according to a chosen scoring function. Despite its simplicity, APE is shown to find prompts that match or surpass human-written prompts. (2) Automatic Prompt Optimization (APO) [2] performs a more directed search compared to APE, which simply proposes and evaluates a bunch of prompts in one pass. We use batches of training data to derive “gradients”—just text-based critiques of the current prompt’s mistakes—that guide edits / improvements to the prompt. Then, we form a recursive feedback loop by: 1. Collecting errors made by the current prompt on the training data. 2. Summarizing these errors via a natural language gradient. 3. Using the gradient to generate several modified versions of the prompt. 4. Selecting the best of the edited prompts. 5. Repeating this process several times. (3) Gradient-free Instructional Prompt Search (GrIPS) [3] uses heuristics to edit prompts instead of prompting an LLM to generate new prompts. All edits–including deletion, swap, paraphrase, and addition–are performed at the phrase level. Only phrases that are previously deleted are considered for addition, and paraphrase operations simply prompt an LLM to paraphrase a phrase. With these edit operations, we can form a prompt optimization strategy by continually editing a set of prompts and selecting those with the best performance. (4) Optimization by Prompting (OPRO) [4] is a generic, gradient-free optimization algorithm that operates by: - Describing an optimization task in natural language. - Showing an optimizer LLM examples of prior solutions to the optimization task along with their objective values. - Asking the optimizer LLM to infer new / better solutions to the problem. - Testing the inferred solutions via an evaluator LLM. One of the most notable applications of OPRO is prompt optimization. The key component of this algorithm is the optimizer LLM, which receives a meta-prompt that contains all information necessary for the LLM to generate a new / better prompt; e.g., prior prompts, prompt performance metrics, few-shot examples of the task, and more. We optimize a prompt by updating this meta-prompt to propose new / better prompts over time. More details. To learn more about prompt optimization algorithms, check out the overview that I just wrote of this topic: https://lnkd.in/g9nxjr6T This writeup outlines most of the literature in this space, including anything from "soft" prompts that are trained via gradients to LLM-based prompt optimizers.

The whole LLM community is underestimating the power of auto-prompt optimization, especially in academics. The effectiveness of prompt engineering (in-context learning) even caught Dr. Manning by surprise. But it is what has made LLMs as prevalent as they are right now. Model fine-tuning with methods such as SFT and DPO (Direct Preference Optimization) is researched far more than auto-prompt optimization and is considered much cooler in the research world. It is true that model fine-tuning is crucial for democratizing LLMs, enabling their adaptation to various end use cases with an OS model without solely relying on proprietary providers. But a huge missing piece is: where does the training dataset come from? In academics, most researchers don’t care about this, as they can use publicly available datasets. But for product teams, you have to make your own datasets. So how can prompt engineering help? Assume we start with a best model (teacher) and an OS (student) you want to optimize. [Teacher and student can also be the same model.] 1️⃣ Leveraging one of the best models, plus a small golden validation and training dataset manually labeled, you can create the training datasets for SFT. [You maximize the performance of existing models and create maybe a 90% accurate training dataset.] 2️⃣ Leveraging an aligned LLM judge, you can create a preference dataset using the student and the teacher. With Step 1 and Step 2, you will optimize your student model to its maximum with minimum human-labeled data. Ideally, you should do this iteratively. But the bottleneck is manual prompt engineering. Every time you fine-tune your target model, you need to go through the manual engineering. Sometimes your app pipeline is complicated, and manual prompting is not even feasible. That is the beauty of auto-prompt optimization for any LLM task pipeline. It closes the loop of optimization with minimal human labeling, relying only on the starter validation and a small training dataset, as in-context learning is essentially the most effective few-shot learning. AdalFlow is a greatly undermined library, but as we mature and combine that with model fine-tuning, the world will be shocked by its power. #artificialintelligence #machinelearning #llms #adalflow