Boosting LLM Reliability Using Representation Engineering

Everyone can prompt engineer. But how do you know that the LLM is actually following your instructions? An upcoming ICLR 2025 paper from University of Cambridge and Apple aimed to answer this very question, unveiling how prompt engineering actually works. The authors explored the internal workings of openweight LLMs (LLaMA-2, Mistral and Phi variants) and revealed a hidden "instruction-following dimension" embedded in the models' representations. By manipulating this dimension, they were able to boost the models’ ability to follow instructions without hurting response quality. Here are the main findings: ✨ LLMs can predict success or failure from the beginning: Surprisingly, LLMs signal early on, in fact right from the *first token of the input prompt* (yes, input), whether they will follow an instruction successfully or fail. This is critical because it opens up opportunities for early intervention in real-time model behavior correction, before the model generates any responses. ✨ There appears to be a hidden instruction-following dimension: The authors pinpointed a specific, linear dimension in the input embedding space that directly correlates with instruction adherence. Through linear probing, they isolated this dimension, showing that it consistently separates instruction-following successes from failures across multiple layers and tokens. ✨ Prompt phrasing has a huge impact: One of the most interesting findings is how sensitive this dimension is to how instructions are phrased. Small changes in prompt wording can swing a failure into a success (you knew that empirically already, but it's now confirmed). This explains why prompt engineering is so effective: it changes how the instruction gets encoded into the model’s internal representation! ✨ Representation engineering boosts adherence: By shifting model representations along this instruction-following dimension, they improved adherence rates significantly, turning failures into successes. This fine-tuning preserved overall task quality, unlike random tweaks that tended to degrade performance. ✨ Generalization across tasks but not across instruction types: While the instruction-following dimension generalizes well across different tasks (e.g., resume writing vs. joke generation), it struggles with unseen instruction types. The internal geometry of the model’s representation space appears to vary significantly between different types of instructions, making it hard to apply the same solution across all instruction forms. Paper in comments. Really insightful one, given also that the results can be extended to interpreting how closed-weight LLMs work to. I'd dare to say that -given the above insights- automated prompt construction (without feedback) is doomed to be more inefficient compared to manual perturbations, as the more meaningful improvements would require a trial-and-error approach and quite a bit of imagination! 🪄

📣 New Paper - ReflCtrl: Controlling LLM Reflection via Representation Engineering We’re excited to share our latest work on improving the efficiency and controllability of reasoning LLMs. Modern reasoning models such as DeepSeek-R1 and QwQ rely on self-reflection to boost accuracy. While powerful, this behavior introduces two major deployment challenges: • ❓ The reflection mechanism is poorly understood — when and why does the model decide to “pause and rethink”? • 💸 Many reflection steps are redundant, significantly increasing token cost with little accuracy gain. In this work, we address both. 🔍 Innovation #1 - A Latent “Reflection Direction” Through representation engineering, we identify a specific latent direction in model activations that governs reflection behavior. This gives us mechanistic insight into when and why reflection is triggered. 📊 Innovation #2 - Stepwise Steering (ReflCtrl) We introduce ReflCtrl, a framework that intervenes only at the beginning of reasoning steps - rather than at every token. This stepwise approach allows precise control over reflection frequency while preserving internal reasoning consistency. 📈 Results On models like QwQ-32B, ReflCtrl reduces reasoning tokens by up to 33.6% with <0.4% accuracy drop. Compared to token-level baselines, it achieves a significantly more stable trade-off between interpretability, efficiency, and performance. This work was led by my PhD students Ge Yan and Chung-En Sun, and presented as a Spotlight at the NeurIPS 2025 Mechanistic Interpretability Workshop. If you’re interested in controllable and efficient reasoning LLMs, check out the paper and code: 🚀 Paper: https://lnkd.in/gGYqKQRG 🚀 Code: https://lnkd.in/gSZzKc5v 🚀 Project Website: https://lnkd.in/gsEKXsng

New research from Intuit AI Research. Agent performance depends on more than just the agent. It also depends on the quality of the tool descriptions it reads. However, tool interfaces are still written for humans, not LLMs. As the number of candidate tools grows, poor descriptions become a real bottleneck for tool selection and parameter generation. As Karpathy has been suggesting, build for AI Agents. This new research introduces Trace-Free+, a curriculum learning framework that teaches models to rewrite tool descriptions into versions that are more effective for LLM agents. The key idea: during training, the model learns from execution traces showing which tool descriptions lead to successful usage. Then, through curriculum learning, it progressively reduces reliance on traces, so at inference time, it can improve tool descriptions for completely unseen tools without any execution history. On StableToolBench and RestBench, the approach shows consistent gains on unseen tools, strong cross-domain generalization, and robustness as candidate tool sets scale beyond 100. Instead of only fine-tuning the agent, optimizing the tool interface itself is a practical and underexplored lever for improving agent reliability.

Can you improve LLM’s reasoning without retraining? Yes. The method, called Representation Engineering, manipulates internal activations during inference to nudge models into more “reasoning-capable” states. Instead of fine-tuning weights, the authors extract residual stream activations associated with high-reasoning prompts. They compute an average “reasoning direction” vector, and then steer the model using this vector during inference. How it Works: 🔹 Identify reasoning-intensive tasks (deductive, inductive, mathematical). 🔹 Log residual activations at specific layers while solving those tasks. 🔹 Use PCA to compute dominant reasoning directions. 🔹 Inject the reasoning vector back into the residual stream of new prompts. This led to Lower entropy, KL divergence shifts, and sharper logits distribution - without model retraining. Take a look at the research paper in the comments below!