Assessing Decision-Making Processes for Robotics

When and How Intelligent Systems Access Knowledge is Fundamental for Agentic 🗯️ Rather than treating retrieval as a simple lookup operation, modern approaches view it as a sophisticated decision-making process that fundamentally shapes how AI systems reason and act. First, the decision of when to retrieve information emerges as a critical cognitive capability in itself. The DeepRAG framework demonstrates that this isn't a simple binary choice but rather a complex decision process that weighs multiple factors including confidence in internal knowledge, potential value of external information, and computational costs. This mirrors human cognitive processes where experts must constantly decide whether to rely on their existing knowledge or consult external sources. Second, the integration of retrieved information represents another sophisticated challenge. The CoAT framework reveals that successful integration requires maintaining coherence with existing reasoning, resolving potential conflicts, and creating meaningful connections between old and new information. This process must be dynamic and adaptive, adjusting to the specific context and requirements of each situation. Third, these insights extend far beyond simple information retrieval, impacting every aspect of agentic systems. Similar principles apply to tool usage, memory management, planning, and knowledge system integration. Each component must make strategic decisions about resource usage and information flow. The mathematical frameworks presented in these papers, particularly the Markov Decision Process approach in DeepRAG and the Chain-of-Associated-Thoughts in CoAT, provide formal mechanisms for understanding and implementing these capabilities. These frameworks enable systems to learn from experience, improving their decision-making about when and how to use different resources. Traditional AI systems often struggle with determining when to rely on internal knowledge versus when to seek external information. The frameworks presented in these papers offer a path forward, showing how systems can develop sophisticated judgment about resource usage while maintaining coherent reasoning processes. The principles of strategic decision-making about information use apply equally to tool selection, memory management, and planning. This suggests a unified approach to building intelligent systems where each component operates with awareness of its resources and limitations. The knowledge graph structure serves as a unifying framework, enabling systems to represent and reason about relationships between different types of information and resources. This integration is crucial for building truly intelligent systems that can adapt to complex, changing environments. By recognizing retrieval as a sophisticated cognitive capability rather than a simple lookup operation, we open new possibilities for building more intelligent and adaptable systems.

When we talk about evaluation in AI agents, it’s tempting to treat it like a checkbox—a set of tests you run after development, or a dashboard you look at before pushing to prod. But in reality, evaluation is something far deeper. It’s not just about checking correctness it’s about building trust in the agent’s ability to reason, adapt, and improve. The more time I spend building autonomous systems, the clearer it becomes: evaluation isn’t something you do to the agent. It’s something the agent needs to do within itself. Think of it like an internal compass—constantly asking: “Did I understand the user’s goal? Is my current plan still valid? Is this tool giving the right feedback? Am I drifting off course?” In that sense, evaluation becomes less of a static test and more of a living, embedded process that runs alongside the agent’s core loop. To get there, we need to start by instrumenting agents with evaluation hooks at every critical junction: during planning, memory updates, tool invocation, interaction with other agents, and response generation. These hooks should feed into lightweight evaluators—some rule-based, some LLM-driven, some trained on reward models—that score and log performance in context. And more importantly, we need to treat those scores as feedback, not just metrics. What’s often missing is the infrastructure to close that loop. So we have to build: 1) agent-centric logging systems that capture decision traces and context windows 2) real-time feedback routers that flag deviations and inconsistencies 3) offline simulators where agents can replay failure scenarios 4) lightweight shadow agents that propose alternative plans for comparison. This becomes even more critical when agents are working in dynamic environments—switching between tools, reasoning across long time horizons, or collaborating with other agents. They can’t just act; they need to reflect in real-time. They need to recognize when something feels off—even if the output looks superficially correct. That’s where things get interesting. We’re starting to think about evaluation not just as a QA layer, but as an integral part of the agent’s intelligence. Just like people develop instincts and checks to catch their own mistakes, agents need that too. Especially in high-stakes domains, where a small misstep can cascade into major failure. We also need to think about how agents evaluate each other, how trust and alignment work in multi-agent systems, and how to build models that know when they don’t know. All of this points toward one big idea: agents need to be eval-native. Evaluation can’t live outside the system—it has to be built into its bones. Because without that constant, self-aware feedback loop, we’re not building autonomous intelligence. We’re building something that only looks smart—until it breaks. https://lnkd.in/eRhQcDc3

Essentially every decision in organizations will be human-AI in some form. So there is little more important than understanding how we configure them for the best outcomes, with clear human accountability throughout. A very interesting paper "From Accuracy to Readiness: Metrics and Benchmarks for Human–AI Decision-Making" makes a very good case for what I would hope is clear: the accuracy of the models is not the most important thing. It is team readiness: the ability of a team to use AI well in improving decision quality and safety. It proposes four domains for measuring team readiness and performance in implementing Humans + AI decision-making, pointing out these are distinct but often conflated. Onboarding is treated as a learning process that should build four concrete competencies: detecting reliability boundaries, calibrating reliance, exercising safe control, and understanding delegation and autonomy. The evaluation is by observing behavior over time. The metrics use detailed decision logs, including of initial human decision, AI predictions, final human decisions, confidence, timestamps. This can be used to evaluate measures like team gain, regret, rollback rate, escalation rate, and more. A lot of the thinking in here is aligned with my current work on Humans + AI decision-making. The reality is few organizations are ready for the depth of these decision evaluation systems. But the principles can be readily implemented relatively easily, in helping organizations successfully transition to the rapid onset of pervasive Humans + AI decision-making.

Reasoning over long horizons would allow robots to generalize better to unseen environments and settings zero-shot. One mechanism for this kind of reasoning would be world models, but traditional video world models still tend to struggle with long horizons, and are very data intensive to train. But what if instead of predicting images about the future, we predicted just the symbolic information necessary for reasoning? Nishanth Kumar tells us about Pixels to Predicates, a method for symbol grounding which allows a VLM to plan sequences of robot skills to achieve unseen goals in previously unseen settings. To find out more, watch episode #44 of RoboPapers with Michael Cho and Chris Paxton now! Abstract: Our aim is to learn to solve long-horizon decision-making problems in complex robotics domains given low-level skills and a handful of short-horizon demonstrations containing sequences of images. To this end, we focus on learning abstract symbolic world models that facilitate zero-shot generalization to novel goals via planning. A critical component of such models is the set of symbolic predicates that define properties of and relationships between objects. In this work, we leverage pretrained vision language models (VLMs) to propose a large set of visual predicates potentially relevant for decision-making, and to evaluate those predicates directly from camera images. At training time, we pass the proposed predicates and demonstrations into an optimization-based model-learning algorithm to obtain an abstract symbolic world model that is defined in terms of a compact subset of the proposed predicates. At test time, given a novel goal in a novel setting, we use the VLM to construct a symbolic description of the current world state, and then use a search-based planning algorithm to find a sequence of low-level skills that achieves the goal. We demonstrate empirically across experiments in both simulation and the real world that our method can generalize aggressively, applying its learned world model to solve problems with a wide variety of object types, arrangements, numbers of objects, and visual backgrounds, as well as novel goals and much longer horizons than those seen at training time. Project Page: https://lnkd.in/e6CWZm8P ArXiV: https://lnkd.in/esnEW5DR Watch now on... Substack: https://lnkd.in/eph7Ew7Y Youtube: https://lnkd.in/eZkkYN5T

As we transition from traditional task-based automation to 𝗮𝘂𝘁𝗼𝗻𝗼𝗺𝗼𝘂𝘀 𝗔𝗜 𝗮𝗴𝗲𝗻𝘁𝘀, understanding 𝘩𝘰𝘸 an agent cognitively processes its environment is no longer optional — it's strategic. This diagram distills the mental model that underpins every intelligent agent architecture — from LangGraph and CrewAI to RAG-based systems and autonomous multi-agent orchestration. The Workflow at a Glance 1. 𝗣𝗲𝗿𝗰𝗲𝗽𝘁𝗶𝗼𝗻 – The agent observes its environment using sensors or inputs (text, APIs, context, tools). 2. 𝗕𝗿𝗮𝗶𝗻 (𝗥𝗲𝗮𝘀𝗼𝗻𝗶𝗻𝗴 𝗘𝗻𝗴𝗶𝗻𝗲) – It processes observations via a core LLM, enhanced with memory, planning, and retrieval components. 3. 𝗔𝗰𝘁𝗶𝗼𝗻 – It executes a task, invokes a tool, or responds — influencing the environment. 4. 𝗟𝗲𝗮𝗿𝗻𝗶𝗻𝗴 (Implicit or Explicit) – Feedback is integrated to improve future decisions.     This feedback loop mirrors principles from: • The 𝗢𝗢𝗗𝗔 𝗹𝗼𝗼𝗽 (Observe–Orient–Decide–Act) • 𝗖𝗼𝗴𝗻𝗶𝘁𝗶𝘃𝗲 𝗮𝗿𝗰𝗵𝗶𝘁𝗲𝗰𝘁𝘂𝗿𝗲𝘀 used in robotics and AI • 𝗚𝗼𝗮𝗹-𝗰𝗼𝗻𝗱𝗶𝘁𝗶𝗼𝗻𝗲𝗱 𝗿𝗲𝗮𝘀𝗼𝗻𝗶𝗻𝗴 in agent frameworks Most AI applications today are still “reactive.” But agentic AI — autonomous systems that operate continuously and adaptively — requires: • A 𝗰𝗼𝗴𝗻𝗶𝘁𝗶𝘃𝗲 𝗹𝗼𝗼𝗽 for decision-making • Persistent 𝗺𝗲𝗺𝗼𝗿𝘆 and contextual awareness • Tool-use and reasoning across multiple steps • 𝗣𝗹𝗮𝗻𝗻𝗶𝗻𝗴 for dynamic goal completion • The ability to 𝗹𝗲𝗮𝗿𝗻 from experience and feedback    This model helps developers, researchers, and architects 𝗿𝗲𝗮𝘀𝗼𝗻 𝗰𝗹𝗲𝗮𝗿𝗹𝘆 𝗮𝗯𝗼𝘂𝘁 𝘄𝗵𝗲𝗿𝗲 𝘁𝗼 𝗲𝗺𝗯𝗲𝗱 𝗶𝗻𝘁𝗲𝗹𝗹𝗶𝗴𝗲𝗻𝗰𝗲 — and where things tend to break. Whether you’re building agentic workflows, orchestrating LLM-powered systems, or designing AI-native applications — I hope this framework adds value to your thinking. Let’s elevate the conversation around how AI systems 𝘳𝘦𝘢𝘴𝘰𝘯. Curious to hear how you're modeling cognition in your systems.

Special thanks to Prof. Ahmed H. Qureshi for the personalized tour of his CORAL Lab at Purdue University. CORAL’s research spans machine learning, robot planning and control, and my personal favorite area, safe human–robot collaboration. Together, these threads tackle one of the hardest problems in robotics today: how autonomous systems can learn, plan, and act effectively while operating alongside people in real, unstructured environments. What stood out to me is how the lab integrates learning and decision-making with explicit attention to safety, interaction, and shared spaces. CORAL explores how robots can reason about uncertainty, model human behavior, and adapt their plans in ways that support collaboration rather than conflict. This includes work on risk-aware planning, learning-based control, and interaction-aware decision-making that directly addresses how robots should behave around and with humans. Several of my favorite papers from the lab dive deeply into these themes, including work on safe and interactive planning, human-aware risk representations, and learning frameworks that support trustworthy collaboration between humans and robots. I’ve shared links to a few of these papers below for anyone who wants to explore further. As robots increasingly leave controlled environments and enter factories, hospitals, warehouses, and public spaces, this kind of research becomes foundational. Autonomy that ignores the human context will struggle to scale. Autonomy that understands and respects it has the potential to truly transform how we work and live. Many thanks again to Ahmed and the CORAL team for the warm welcome and the great conversations (remember: replace the banana with a beer bottle for social impact! 😉 ). It was energizing to see research that so clearly connects theory, algorithms, and real-world human impact. Purdue Computer Science ---- For those interested in going deeper, here are a few of my favorite papers from the CORAL Lab that really capture the breadth and impact of their work: 🔹 Safe and interactive planning for human–robot collaboration https://lnkd.in/eMMqED3r https://lnkd.in/eQJtSinY 🔹 Risk-aware representations and decision-making around humans https://lnkd.in/ep4dRPHM 🔹 Learning and control frameworks that enable safe, trustworthy interaction https://lnkd.in/eVjcQBpa https://lnkd.in/eSH-_CrV These papers do a great job of connecting learning, planning, and control with the realities of shared human–robot environments. Highly recommend a read if you’re working in robotics, autonomy, or human–robot interaction.

Oxford proves two heads are indeed better than one in robot teleoperation Teleoperating robots in safety critical, dynamic environments often leaves single operators overwhelmed due to uncertainty, cognitive overload, and task complexity. Researchers addressed this challenge through collaborative decision making in teleoperation, and here are their findings: 1) Using Maximum Confidence Slating (MCS), decisions guided by the most confident operator significantly 𝗶𝗺𝗽𝗿𝗼𝘃𝗲𝗱 𝗮𝗰𝗰𝘂𝗿𝗮𝗰𝘆 𝗯𝗲𝘆𝗼𝗻𝗱 𝘁𝗵𝗲 𝗯𝗲𝘀𝘁 𝗶𝗻𝗱𝗶𝘃𝗶𝗱𝘂𝗮𝗹'𝘀 𝗽𝗲𝗿𝗳𝗼𝗿𝗺𝗮𝗻𝗰𝗲. 2) Accuracy gains from joint decision making were 𝗵𝗶𝗴𝗵𝗲𝘀𝘁 𝘄𝗵𝗲𝗻 𝗯𝗼𝘁𝗵 𝗼𝗽𝗲𝗿𝗮𝘁𝗼𝗿𝘀 𝗵𝗮𝗱 𝘀𝗶𝗺𝗶𝗹𝗮𝗿 𝘀𝗸𝗶𝗹𝗹 𝗹𝗲𝘃𝗲𝗹𝘀; large skill gaps actually diminished joint accuracy. 3) 𝗔𝗰𝗰𝘂𝗿𝗮𝘁𝗲 𝗰𝗼𝗻𝗳𝗶𝗱𝗲𝗻𝗰𝗲 𝗰𝗮𝗹𝗶𝗯𝗿𝗮𝘁𝗶𝗼𝗻 in at least one operator significantly enhanced joint decision making performance. 4) 𝗛𝗶𝗴𝗵𝗲𝗿 𝗼𝘃𝗲𝗿𝗮𝗹𝗹 𝗱𝘆𝗮𝗱𝗶𝗰 𝗰𝗼𝗻𝗳𝗶𝗱𝗲𝗻𝗰𝗲 𝗰𝗮𝗹𝗶𝗯𝗿𝗮𝘁𝗶𝗼𝗻 correlated strongly with improved decision accuracy. I'm interested in how they implemented the psychological phenomenon, 'two heads are better than one', into robotic teleoperation. They utilized MCS method, selecting the robot with the highest confidence.  If you haven't noticed yet, this phenomenon is utilized in multiple fields. In medical diagnostics, where pooling independent judgments leads to increased diagnostic accuracy. In 'self-consistency' methods utilized in the popular LLMs today. Love to see researchers being able to converge another distinct field into robotic teleoperation. Congratulations to An Nguyen, Raunak Bhattacharyya, Clara Colombatto, Steve Fleming, Ingmar Posner, and Nick Hawes! Paper link: https://lnkd.in/gscsY3Eb I post the latest and interesting developments in robotics, 𝗳𝗼𝗹𝗹𝗼𝘄 𝗺𝗲 𝘁𝗼 𝘀𝘁𝗮𝘆 𝘂𝗽𝗱𝗮𝘁𝗲𝗱!