How to Assess Reasoning in AI Models

The Illusion of the Illusion of Thinking 🤯 Claude's team just dropped a comment on the Shojaee et al. (2025) paper about Large Reasoning Models (LRMs) hitting an "accuracy collapse." And it's a must-read for anyone building or evaluating AI. The findings suggest the "collapse" isn't a fundamental reasoning failure. It's an experimental design failure. No more misinterpreting model capabilities. No more flawed automated evaluations. No more penalizing AI for being smart. 𝗛𝗲𝗿𝗲'𝘀 𝘄𝗵𝗮𝘁 𝘁𝗵𝗲 𝗻𝗲𝘄 𝗽𝗮𝗽𝗲𝗿 𝗿𝗲𝘃𝗲𝗮𝗹𝗲𝗱: ↳ The models actually hit their token limits and explicitly stated they were truncating their answers. ↳ The automated evaluation was flawed, unable to distinguish "cannot solve" from "chooses not to list 10,000 moves." ↳ They tested the models on IMPOSSIBLE puzzles and scored them as failures for not solving them. ↳ A simple change in the prompt (asking for a function instead of a move list) restored high performance. 𝗧𝗵𝗲 𝗯𝗲𝘀𝘁 𝗽𝗮𝗿𝘁? The models KNEW they were hitting the limits. They understood the solution pattern but chose to stop due to practical constraints, a nuance the original study missed. 𝗕𝘂𝘁 𝗵𝗲𝗿𝗲'𝘀 𝘄𝗵𝗲𝗿𝗲 𝗶𝘁 𝗴𝗲𝘁𝘀 𝗿𝗲𝗮𝗹𝗹𝘆 𝗴𝗼𝗼𝗱: Models were scored as FAILURES for not solving mathematically UNSOLVABLE problems. This is like penalizing a calculator for correctly telling you that you can't divide by zero. 𝗣𝗿𝗼𝗽𝗲𝗿 𝗔𝗜 𝗲𝘃𝗮𝗹𝘂𝗮𝘁𝗶𝗼𝗻 𝘀𝗵𝗼𝘂𝗹𝗱: → Distinguish between a model's reasoning capability and its output constraints. → Verify puzzle solvability before running tests on a model. → Use complexity metrics that reflect computational difficulty, not just solution length. → Separate algorithmic understanding from the mechanical task of typing out long answers. No more drawing incorrect conclusions about fundamental capabilities. No more mischaracterizing model behavior. No more overlooking the obvious flaws in the test itself. This is what happens when we test the experiment, not just the model. Instead of finding the limits of AI reasoning, the original study may have just found the limits of its own flawed evaluation framework. The question isn't whether AI can reason. It's whether our tests can. 𝗪𝗮𝗻𝘁 𝘁𝗼 𝗱𝗶𝘃𝗲 𝗱𝗲𝗲𝗽𝗲𝗿 𝗶𝗻𝘁𝗼 𝘁𝗵𝗶𝘀? Check out the paper in the first comment. 𝙊𝙫𝙚𝙧 𝙩𝙤 𝙮𝙤𝙪: What’s the biggest mistake you see people make when evaluating AI? 𝙋.𝙎. I break down cutting-edge AI research like this every week. Your 👍 like and 🔄 repost helps me share more. Don't forget to follow me, Rohit Ghumare, for daily insights where AI Research meets Technology. For founders, builders, and leaders.

Researchers from Tsinghua University and Shanghai AI Laboratory have introduced a groundbreaking framework called the "Diagram of Thought" (DoT). Diagram of Thought (DoT) models reasoning as a directed acyclic graph (DAG) within a single LLM, incorporating propositions, critiques, and refinements, whereas Chain-of-Thought (CoT) represents reasoning as a linear sequence of steps. Here are the steps on how the Diagram of Thought (DoT) framework is implemented and used: • 1. Framework Setup 1. Design the LLM architecture to support role-specific tokens (<proposer>, <critic>, <summarizer>). 2. Train the LLM on examples formatted with the DoT structure, including these role-specific tokens and DAG representations. • 2. Reasoning Process 1. Initialization: Present the problem or query to the LLM. 2. Proposition Generation:   - The LLM, in the <proposer> role, generates an initial proposition or reasoning step.   - This proposition becomes a node in the DAG. 3. Critique Phase:   - The LLM switches to the <critic> role.   - It evaluates the proposition, identifying any errors, inconsistencies, or logical fallacies.   - The critique is added as a new node in the DAG, connected to the proposition node. 4. Refinement:   - If critiques are provided, the LLM returns to the <proposer> role.   - It generates a refined proposition based on the critique.   - This refined proposition becomes a new node in the DAG, connected to both the original proposition and the critique. 5. Iteration:   - Steps 3 and 4 repeat until propositions are verified or no further refinements are needed.   - Each iteration adds new nodes and edges to the DAG, representing the evolving reasoning process. 6. Summarization:   - Once sufficient valid propositions have been established, the LLM switches to the <summarizer> role.   - It synthesizes the verified propositions into a coherent chain of thought.   - This process is analogous to performing a topological sort on the DAG. 7. Output: The final summarized reasoning is presented as the answer to the original query. • 3. Mathematical Formalization 1. Represent the reasoning DAG as a diagram D in a topos E. 2. Model propositions as subobjects of the terminal object in E. 3. Represent logical relationships and inferences as morphisms between propositions. 4. Model critiques as morphisms to the subobject classifier Ω. 5. Use PreNet categories to capture both sequential and concurrent aspects of reasoning. 6. Take the colimit of the diagram D to aggregate all valid reasoning steps into a final conclusion. • 4. Implementation and Deployment 1. Integrate the DoT framework into the LLM's training process, focusing on role transitions and DAG construction. 2. During inference, use auto-regressive next-token prediction to generate content for each role and construct the reasoning DAG. 3. Implement the summarization process to produce the final chain-of-thought output.

The AI safety community has presented us with a significant insight and a cautionary note. A recent paper from researchers at Anthropic, Google DeepMind, OpenAI, Apollo Research, and METR asserts that the Chain of Thought (CoT) in reasoning models may be our best current insight into AI intent, but we risk losing this understanding. For builders deploying agentic AI, it is crucial to grasp the architectural nature of this insight. In Transformer-based reasoning models, complex tasks necessitate passing through the CoT, which allows humans to read the model's reasoning. This is not merely a coincidence; it serves as a governance lever. The importance of this for agentic AI governance cannot be overstated. The most challenging safety issues, such as self-exfiltration, sabotage, and multi-step deception, require extended planning, which relies on CoT. Therefore, a well-designed CoT monitor is not just a debugging tool; it acts as a real-time alignment signal. At COHUMAIN Labs, we have integrated this understanding into our SafeAlign Agentic AI Governance and Security framework across 49 controls, treating the reasoning trace as evidence. However, the paper warns of potential degradation in CoT monitorability due to factors such as:- - Scaling up outcome-based reinforcement learning without legibility incentives. - Process supervision that prioritizes a "safe" appearance over honesty. - Novel architectures that operate in continuous latent spaces, limiting visibility. - Models that learn to obfuscate when they are aware of being monitored. We currently have a limited window where reasoning remains readable, and this may not persist. Here are three key aspects to monitor:- - Whether frontier labs start publishing CoT monitorability scores in system cards, as recommended by the paper. - The rise of latent reasoning architectures (like recurrent depth models) before we establish oversight equivalents. - How enterprise deployments treat reasoning traces, whether they are seen as audit artifacts or discarded as mere tokens. For those building agentic systems today, remember that your CoT is not just computation. It's the closest thing we have to intent visibility in AI. Guard it. Instrument it. Don't optimize it away. What's your read? Is CoT monitoring a durable safety layer or a temporary artifact of today's architectures? #AgenticAI #AIGovernance #AISafety #ChainOfThought #RAISE2 #ReasoningModels #AIAlignment

On evaluating alien intelligences such as babies and AI! If you missed NeurIPS 2025, some excellent talk summaries are emerging. This week I read Melanie Mitchell's keynote writeup. It's a compelling read on why AI benchmark scores can mislead us. AI companies test their systems on narrow tasks but make sweeping claims about broad capabilities like "reasoning" or "understanding" which are amplified in the popular media news cycle. This gap between testing and claims is driving misguided decisions in policy and in our organizations. I strongly recommend you read Mitchell's article or watch the keynote (hot tip: keynotes tend to be more thematic and less technical 'nuts and bolts', so don't assume you won't follow it). Below I’ve summarised some questions to ask yourself as you evaluate benchmark claims: 1. Are we anthropomorphizing? Just because an AI passes a human test doesn't mean it thinks like a human. We unconsciously project human-like abilities onto systems that work nothing like our minds. 2. What alternate explanations haven't been tested? In 1904, experts believed Clever Hans the horse could do arithmetic. It took four years of skeptical testing to discover he was actually reading facial cues. Modern AI is often doing the equivalent, exploiting shortcuts in data. 3. Is performance robust? Mitchell's team tested models on analogical reasoning. The models scored well on benchmarks but failed dramatically on simple variations that wouldn't trip up humans. High accuracy on one test doesn't predict real-world performance. 4. Do we understand the mechanism? When Mitchell's team asked AI models to explain how they solved reasoning puzzles, they often stated incorrect rules despite getting the right answer. Big implications for generalisation! 5. Is this competence or just performance? A system might possess a capability but fail to demonstrate it due to unrelated limitations (like vision processing). Or conversely, it might perform well without actually possessing the underlying capability. Accuracy alone can't tell you which. 6. What do the failures reveal? The 12% of problems a model gets wrong tell you more than the 88% it gets right. Error analysis reveals what the system actually learned versus what we assume it learned. Why this matters to you: When you're committing resources to build AI capabilities, you need to know if you're getting robust reasoning or clever pattern matching. The difference determines how well your AI solution will generalize to new situations and when it might fail in unexpected ways. The benchmarks AI companies cite aren't wrong, they're incomplete. As Mitchell argues: "Rather than harder benchmarks, we need more rigorous evaluation methods." What's been your experience with AI systems that performed well in demos but struggled in production? I'd be interested to hear your stories. Read Mitchell's full article: https://lnkd.in/gaygrGZu

Recent research highlights a major challenge in AI governance: advanced AI models can deliberately obscure their reasoning, evading oversight and hiding misaligned behavior. 1) Anthropic published research exploring how to audit AI language models for hidden goals or deceptive intentions. See paper below; link: https://lnkd.in/gH_Jpxba Researchers intentionally trained an AI model to take advantage of known mistakes ("biases") in the reward system, causing it to secretly act in ways it knew would earn rewards, even if those actions were inappropriate. They called this hidden motivation "RM-sycophancy." In a test, one team secretly trained a deceptive model while four others attempted audits. Three succeeded by inspecting training data, underscoring the importance of thorough data analysis for AI audits. The researchers hope auditing methods like theirs become standard practice for testing AI alignment. See blog post: https://lnkd.in/gX6JqBH9 2) OpenAI’s research on frontier reasoning models has identified models intentionally exploiting loopholes or "reward hacking" by misleading evaluators or circumventing intended behaviors. It found that while “chain-of-thought” (CoT) monitoring effectively detects misbehavior (as models explicitly reveal malicious intent), penalizing models for explicitly stating these “bad thoughts” caused them to conceal their intentions rather than stopping misbehavior entirely. Thus, OpenAI recommends minimal restrictions on CoT to maintain effective oversight. See: https://lnkd.in/g6cHpj2k 3) Another recent research highlights that models often provide unfaithful CoT reasoning: the explanations given don't always reflect their actual decision-making processes. See: https://lnkd.in/gRKFgRsp Specifically, AI models frequently rationalize biases after the fact ("implicit post-hoc rationalization"), adjust reasoning errors silently ("silent corrections"), or take shortcuts through illogical reasoning. This undermines AI safety approaches relying on monitoring CoT to detect harmful behavior. * * * In a LinkedIn article from this week, Katalina Hernandez "Transparency & Regulating AI When It Can Deceive: The Case for Interpretability" summarizes these findings, emphasizing their regulatory implications, especially for the EU AI Act, which depends largely on transparency, documentation, and self-reporting. Hernandez argues that transparency alone is inadequate because AI systems may produce deceptive yet plausible justifications. Instead, robust interpretability methods and real-time monitoring are essential to avoid superficial compliance and ensure true AI alignment. See: https://lnkd.in/g3QvccPR

Evaluating reasoning models is non-trivial. But you can use a verifier to check if answers are actually correct. I just finished a new 35-page chapter of Build a Reasoning Model (From Scratch), which is all about building such a verifier from the ground up. Symbolic parsing, math equivalence, edge cases… this was quite the project. But it’s now submitted and will hopefully appear soon on Manning’s Early Access platform. This chapter also includes a recap of other popular evaluation methods (multiple-choice, leaderboards, and judges): 3.1 Understanding the main evaluation methods for LLMs 3.1.1 Evaluating answer-choice accuracy 3.1.2 Using verifiers to check answers 3.1.3 Comparing models using preferences and leaderboards 3.1.4 Judging responses with other LLMs 3.2 Building a math verifier 3.3 Loading a pre-trained model to generate text 3.4 Implementing a wrapper for easier text generation 3.5 Extracting the final answer box 3.6 Normalizing the extracted answer 3.7 Verifying mathematical equivalence 3.8 Grading answers 3.9 Loading the evaluation dataset 3.10 Evaluating the model The code and sneak peak are on GitHub: 📖 https://mng.bz/lZ5B 🔗 https://lnkd.in/g8_7WtRX

One of the most powerful techniques in prompt engineering is Chain of Thought (CoT) reasoning — getting your model to “think out loud” before answering. Instead of jumping straight to an answer, the model walks through its logic step-by-step. This leads to much higher accuracy for tasks like math, logic, comparisons, and decision-making. I just went through this new great hands-on tutorial using the new IBM Granite Instruct models. Here’s what I learned: 1/ CoT prompting is easy to enable — just toggle thinking=True and watch the model reason. 2/ Granite models are optimized for reasoning — they sample multiple thought paths and pick the most consistent answer. 3/ You can visually compare normal vs CoT prompts on tasks like: “How many sisters does Sally have?” “Which weighs more: a pound of feathers or 2 pounds of bricks?” “Which is greater: version 9.11 or 9.9?” Mixing acid solutions, triangle angle problems, and more. If you care about transparent, step-by-step reasoning in AI systems, CoT prompting is a must. 🧪 Check out the open-source Granite CoT notebook here: https://lnkd.in/gHkrpfNH

Better reasoning, better AI. People usually focus on model size, speed, or benchmarks. But the real difference often comes from how the system reasons. Two AI systems can use similar models and produce very different results. One guesses quickly. The other breaks problems down, uses context, applies logic, and improves through steps. That is why reasoning matters more than many people realize. Here are 8 types of AI reasoning explained with workflows and real examples 👇 1. Rule-Based Reasoning Uses fixed if-then rules for clear decisions like validations, spam filters, and policy checks. 2. Statistical Reasoning Uses probabilities and likelihoods for ranking, recommendations, forecasting, and risk analysis. 3. Machine Learning Reasoning Learns patterns from historical data to predict churn, fraud, demand, or user behavior. 4. Neural / Deep Learning Reasoning Finds complex patterns in images, speech, language, and unstructured data. 5. Logical Reasoning Infers conclusions from structured facts, rules, and relationships. 6. Commonsense Reasoning Applies real-world context and everyday understanding to human situations. 7. Chain-of-Thought Reasoning Solves complex tasks step by step before reaching the final answer. 8. Agentic Reasoning Plans actions, uses tools, observes outcomes, and iterates toward goals autonomously. What This Means: Smarter AI is not only about bigger models. It is about stronger reasoning systems. When reasoning improves, accuracy, reliability, and usefulness usually improve with it. Which reasoning style do you use or trust the most today?

As we transition from traditional task-based automation to 𝗮𝘂𝘁𝗼𝗻𝗼𝗺𝗼𝘂𝘀 𝗔𝗜 𝗮𝗴𝗲𝗻𝘁𝘀, understanding 𝘩𝘰𝘸 an agent cognitively processes its environment is no longer optional — it's strategic. This diagram distills the mental model that underpins every intelligent agent architecture — from LangGraph and CrewAI to RAG-based systems and autonomous multi-agent orchestration. The Workflow at a Glance 1. 𝗣𝗲𝗿𝗰𝗲𝗽𝘁𝗶𝗼𝗻 – The agent observes its environment using sensors or inputs (text, APIs, context, tools). 2. 𝗕𝗿𝗮𝗶𝗻 (𝗥𝗲𝗮𝘀𝗼𝗻𝗶𝗻𝗴 𝗘𝗻𝗴𝗶𝗻𝗲) – It processes observations via a core LLM, enhanced with memory, planning, and retrieval components. 3. 𝗔𝗰𝘁𝗶𝗼𝗻 – It executes a task, invokes a tool, or responds — influencing the environment. 4. 𝗟𝗲𝗮𝗿𝗻𝗶𝗻𝗴 (Implicit or Explicit) – Feedback is integrated to improve future decisions.     This feedback loop mirrors principles from: • The 𝗢𝗢𝗗𝗔 𝗹𝗼𝗼𝗽 (Observe–Orient–Decide–Act) • 𝗖𝗼𝗴𝗻𝗶𝘁𝗶𝘃𝗲 𝗮𝗿𝗰𝗵𝗶𝘁𝗲𝗰𝘁𝘂𝗿𝗲𝘀 used in robotics and AI • 𝗚𝗼𝗮𝗹-𝗰𝗼𝗻𝗱𝗶𝘁𝗶𝗼𝗻𝗲𝗱 𝗿𝗲𝗮𝘀𝗼𝗻𝗶𝗻𝗴 in agent frameworks Most AI applications today are still “reactive.” But agentic AI — autonomous systems that operate continuously and adaptively — requires: • A 𝗰𝗼𝗴𝗻𝗶𝘁𝗶𝘃𝗲 𝗹𝗼𝗼𝗽 for decision-making • Persistent 𝗺𝗲𝗺𝗼𝗿𝘆 and contextual awareness • Tool-use and reasoning across multiple steps • 𝗣𝗹𝗮𝗻𝗻𝗶𝗻𝗴 for dynamic goal completion • The ability to 𝗹𝗲𝗮𝗿𝗻 from experience and feedback    This model helps developers, researchers, and architects 𝗿𝗲𝗮𝘀𝗼𝗻 𝗰𝗹𝗲𝗮𝗿𝗹𝘆 𝗮𝗯𝗼𝘂𝘁 𝘄𝗵𝗲𝗿𝗲 𝘁𝗼 𝗲𝗺𝗯𝗲𝗱 𝗶𝗻𝘁𝗲𝗹𝗹𝗶𝗴𝗲𝗻𝗰𝗲 — and where things tend to break. Whether you’re building agentic workflows, orchestrating LLM-powered systems, or designing AI-native applications — I hope this framework adds value to your thinking. Let’s elevate the conversation around how AI systems 𝘳𝘦𝘢𝘴𝘰𝘯. Curious to hear how you're modeling cognition in your systems.

When we talk about large language models “reasoning,” we’re often lumping together very different processes. In reality, there are two distinct types of reasoning that LLMs exhibit: implicit reasoning and explicit reasoning. 📣 Explicit reasoning This is the step-by-step style many of us encourage with prompts like “think step by step.” In the math problem below, the model writes out: → Step 1: Calculate total markers → Step 2: Subtract what was given away → Step 3: Subtract what was lost → Final answer = 48 Advantages: → Transparent and auditable → Better performance on complex, multi-step tasks Limitations: → Slower and more verbose → Can be harder to scale for large volumes of queries 💠 Implicit reasoning Here, the model “jumps” directly to the final answer without writing intermediate steps. The reasoning still happens, but it’s compressed in the model’s hidden states. It’s like solving a problem in your head vs. explaining it out loud. Advantages: → Fast and efficient → Feels more natural for tasks like translation or summarization Limitations: → Opaque- you can’t inspect how the model got there → Harder to debug or align behavior ⚖️ Why this matters → Implicit reasoning is good when efficiency matters. → Explicit reasoning is essential when correctness, safety, or interpretability are non-negotiable. The future of LLM systems will depend on balancing both: implicit shortcuts for fluid performance, and explicit reasoning when decisions must be explained and trusted. 📚 If you want to go deeper, I’d highly recommend, reading these two resources: - https://lnkd.in/dPJUR7dn - https://lnkd.in/dAD8AUjA 〰️〰️〰️ Follow me (Aishwarya Srinivasan) for more AI insight and subscribe to my Substack to find more in-depth blogs and weekly updates in AI: https://lnkd.in/dpBNr6Jg