Key Training Data for Service Robot Development

🦾 Great milestone for open-source robotics: pi0 & pi0.5 by Physical Intelligence are now on Hugging Face, fully ported to PyTorch in LeRobot and validated side-by-side with OpenPI for everyone to experiment with, fine-tune & deploy in their robots! π₀.₅ is a Vision-Language-Action model which represents a significant evolution from π₀ to address a big challenge in robotics: open-world generalization. While robots can perform impressive tasks in controlled environments, π₀.₅ is designed to generalize to entirely new environments and situations that were never seen during training. Generalization must occur at multiple levels: - Physical Level: Understanding how to pick up a spoon (by the handle) or plate (by the edge), even with unseen objects in cluttered environments - Semantic Level: Understanding task semantics, where to put clothes and shoes (laundry hamper, not on the bed), and what tools are appropriate for cleaning spills - Environmental Level: Adapting to "messy" real-world environments like homes, grocery stores, offices, and hospitals The breakthrough innovation in π₀.₅ is co-training on heterogeneous data sources. The model learns from: - Multimodal Web Data: Image captioning, visual question answering, object detection - Verbal Instructions: Humans coaching robots through complex tasks step-by-step - Subtask Commands: High-level semantic behavior labels (e.g., "pick up the pillow" for an unmade bed) - Cross-Embodiment Robot Data: Data from various robot platforms with different capabilities - Multi-Environment Data: Static robots deployed across many different homes - Mobile Manipulation Data: ~400 hours of mobile robot demonstrations This diverse training mixture creates a "curriculum" that enables generalization across physical, visual, and semantic levels simultaneously. Huge thanks to the Physical Intelligence team & contributors Model: https://lnkd.in/eAEr7Yk6 LeRobot: https://lnkd.in/ehzQ3Mqy

First empirical evidence that VLA models scale with massive real-world robot data. VLA foundation models promise robots that can follow natural language instructions and adapt to new tasks quickly. However, the field has lacked comprehensive studies on how performance actually scales with real-world data. This new research introduces LingBot-VLA, a Vision-Language-Action foundation model trained on approximately 20,000 hours of real-world manipulation data from 9 dual-arm robot configurations. Scaling pre-training data from 3,000 hours to 20,000 hours improves downstream success rates consistently, with no signs of saturation. More data still helps. The architecture uses a Mixture-of-Transformers design that couples a pre-trained VLM (Qwen2.5-VL) with an action expert through shared self-attention. This allows high-dimensional semantic priors to guide action generation while avoiding cross-modal interference. On the GM-100 benchmark spanning 100 tasks across 3 robotic platforms with 22,500 evaluation trials, LingBot-VLA achieves 17.30% success rate and 35.41% progress score, outperforming π0.5 (13.02% SR, 27.65% PS), GR00T N1.6 (7.59% SR, 15.99% PS), and WALL-OSS (4.05% SR, 10.35% PS). In simulation on RoboTwin 2.0, the model reaches 88.56% success rate in clean scenes and 86.68% in randomized environments, beating π0.5 by 5.82% and 9.92% respectively. Training efficiency matters for scaling. Their optimized codebase achieves 261 samples per second per GPU on an 8-GPU setup, representing a 1.5-2.8× speedup over existing VLA codebases like StarVLA, OpenPI, and DexBotic. Data efficiency is equally impressive: with only 80 demonstrations per task, LingBot-VLA outperforms π0.5 using the full 130-demonstration set. This is the first empirical demonstration that VLA performance continues scaling with more real-world robot data without saturation, providing a clear roadmap for building more capable robotic foundation models.

TOUCAN: Synthesizing 1.5M Tool-Agentic Data from Real-World MCP Environments ... The Dataset That Teaches AI Agents to Actually Use Tools .... Ever wondered why your AI assistant sometimes fails spectacularly when you ask it to use multiple tools together? The problem isn't the AI - it's the training data. Most existing datasets for teaching AI agents to use tools suffer from three critical flaws: they're too small, use fake tool responses, or only cover simple single-tool scenarios. Real-world tasks require agents to coordinate multiple tools, handle failures gracefully, and engage in back-and-forth conversations. 👉 What TOUCAN brings to the table Researchers from University of Washington and MIT-IBM Watson AI Lab just released TOUCAN - a dataset containing 1.5 million tool-agent trajectories built from nearly 500 real-world Model Context Protocol (MCP) servers. Unlike previous attempts, TOUCAN captures authentic tool interactions across diverse domains: from cryptocurrency analysis to weather forecasting, from code execution to document processing. Each trajectory shows the complete decision-making process - when to call tools, how to handle errors, and how to synthesize results. 👉 The technical breakthrough The researchers developed a five-stage pipeline that automatically generates realistic scenarios requiring multiple tools working together. They used actual MCP servers (not simulations) to ensure tool responses reflect real-world behavior. Three key extensions make the dataset especially valuable: - Irrelevance scenarios where agents must recognize when tools can't help - Multi-turn conversations that mirror actual user interactions - Persona-based diversification that creates varied contexts for the same underlying tasks 👉 Performance that speaks volumes Models fine-tuned on TOUCAN outperform much larger closed-source models on standard benchmarks. A 32B parameter model trained on this data beats models with 671B parameters on real-world MCP tasks. The dataset is publicly available, giving the open-source community access to the kind of high-quality training data that was previously available only to large tech companies. This represents a significant step toward AI agents that can reliably handle complex, multi-step tasks in production environments.

1. Scan 2. Demo 3. Track 4. Render 5. Train models 6. Deploy What if robots could learn new tasks from just a smartphone scan and a single human demonstration, without needing physical robots or complex simulations? [⚡Join 2400+ Robotics enthusiasts - https://lnkd.in/dYxB9iCh] A paper by Justin Yu, Letian (Max) Fu, Huang Huang, Karim El-Refai, Rares Andrei Ambrus, Richard Cheng, Muhammad Zubair Irshad, and Ken Goldberg from the University of California, Berkeley and Toyota Research Institute Introduces a scalable approach for generating robot training data without dynamics simulation or robot hardware. "Real2Render2Real: Scaling Robot Data Without Dynamics Simulation or Robot Hardware" • Utilises a smartphone-captured object scan and a single human demonstration video as inputs • Reconstructs detailed 3D object geometry and tracks 6-DoF object motion using 3D Gaussian Splatting • Synthesises thousands of high-fidelity, robot-agnostic demonstrations through photorealistic rendering and inverse kinematics • Generates data compatible with vision-language-action models and imitation learning policies • Demonstrates that models trained on this data can match the performance of those trained on 150 human teleoperation demonstrations • Achieves a 27× increase in data generation throughput compared to traditional methods This approach enables scalable robot learning by decoupling data generation from physical robot constraints. It opens avenues for democratising robot training data collection, allowing broader participation using accessible tools. If robots can be trained effectively without physical hardware or simulations, how will this transform the future of robotics? Paper: https://lnkd.in/emjzKAyW Project Page: https://lnkd.in/evV6UkxF #RobotLearning #DataGeneration #ImitationLearning #RoboticsResearch #ICRA2025

We continue our developments in physical AI and humanoid robots. Here's a behind-the-scenes view of what it takes to create effective training data to train the AI models controlling our robots. At Cambridge Tech Week last month I gave an overview of Seizing AI Advantage in the emerging field of Physical AI. If you missed it, you can watch it back here: https://lnkd.in/etTdEvWx One of the topics I mentioned was the scarcity of available training data for training robots to manipulate small objects. To solve this, we have installed our own motion-capture rigs and created tele-opeartion pipelines so that we can create multiple scenarios of the motion sequences we want our robots to learn. In this video you can see two of our team controlling two of the robots in our lab. Dom is wearing special gloves that are being tracked by the motion-capture rig above him and then directly control the movement of both a simulated robot on the video screen and the physical robot in real-time. At the same time, Cuong is using the AppleVision Pro headset which gives him the robot-eye-view and converts his hand movements into real-time instructions for the robot. All of this allows us to adapt the latest robotics foundation models (we're using NVIDIA Groot) to perform specific actions in a range of settings. That could be moving packages in a logistics setting, having a boxing match or perhaps even doing cartwheels! Great work guys! More to come... #PhysicalAI #HumanoidRobots #AI #Innovation Kary Bheemaiah Sally Epstein Riccardo Secoli Dominic Rugg-Gunn Cuong Kasperzyk John Robins Mat Gilbert, PhD Ali Shafti Cambridge Consultants

Teleoperation data is the endgame If you had infinite quantities of any robotics data type, what would your training recipe look like? Teleop data is the gold layer. Teleop data provides on-embodiment data, in real environments, with real physics. It's the complete package. Internet video, ego data, and UMI all play a role, but as robots are deployed at scale in real world environments, the value from teleoperation data will explode. The bottleneck in teleop data is that most companies don't have the infrastructure to capture it at quality and scale. That's what we built Adamo's Data platform around. Every teleoperation session becomes a structured, exportable training example: synchronized video, trajectories, full telemetry, and a Python SDK that lets you pull it straight into your training pipeline. A lot of people feel they should avoid teleoperation to reach autonomy. We should in fact embrace it!

𝗠𝗼𝗿𝗲 𝗱𝗮𝘁𝗮 𝗱𝗼𝗲𝘀 𝗻𝗼𝘁 𝗳𝗶𝘅 𝗯𝗮𝗱 𝗱𝗮𝘁𝗮. 𝟭𝟬,𝟬𝟬𝟬 𝗶𝗱𝗲𝗻𝘁𝗶𝗰𝗮𝗹 𝗱𝗲𝗺𝗼𝗻𝘀𝘁𝗿𝗮𝘁𝗶𝗼𝗻𝘀 𝗴𝗲𝗻𝗲𝗿𝗮𝗹𝗶𝘀𝗲 𝘄𝗼𝗿𝘀𝗲 𝘁𝗵𝗮𝗻 𝗮 𝗳𝗲𝘄 𝗵𝘂𝗻𝗱𝗿𝗲𝗱 𝗱𝗶𝘃𝗲𝗿𝘀𝗲 𝗼𝗻𝗲𝘀. The standard approach: collect 10,000 pick-and-place examples and assume the model will generalise. Then deployment fails because every demonstration was collected:  • Under the same warehouse lighting  • With the same 5 object types  • On the same table surface The model learned to pick red blocks under fluorescent lighting on a white table. It did not learn to pick objects under varying conditions. Large datasets feel like they should generalise. But dataset size and dataset diversity are independent. You can have 10,000 demonstrations with zero environment diversity, or a few hundred that span lighting conditions, object geometries, and surface textures. What works instead: demonstrations collected across the conditions your deployment will encounter. If your robot operates in warehouses with skylights, your training data needs morning sun, afternoon shadows, and evening artificial lighting. If your robot handles 50 object types, your training data cannot show only 5. This is harder than collecting 10,000 identical demonstrations. Recording data across diverse conditions requires intentional data collection strategy, not just running the robot repeatedly in the same setup. Diverse data collection requires tracking which demonstrations were collected under which conditions. Without metadata tagging, you cannot verify your dataset actually covers your deployment distribution. You cannot filter by lighting condition, object type, or success/failure mode. You cannot diagnose whether your model failed because it never saw dim lighting or because it never saw cylindrical objects. If you are measuring dataset size without measuring dataset diversity, you are collecting data blind. Metadata-tagged data collection is core to how Neuracore is built. #Neuracore #RobotLearning

The robotics community has a name for it now: the 100,000-year data gap. You can't scrape robot training data the way you scrape text. It has to be built. And the two options most teams have — teleoperation and hand-authored simulation — are either too expensive to scale or too synthetic to trust at deployment. Here's the part that kept me up at night: Every time a robot hesitates, clips something, or triggers a safety stop in the real world, that's ground-truth data. It's the exact edge case your sim never generated. It has trajectory, context, spatial geometry, failure signature. And in the current workflow, it gets reset and discarded. The failure repeats. The training set stays thin. The sim-to-real gap stays wide. We built Reconstructiv to close that loop. When an incident happens on a real fleet, we detect it, capture the logs and video automatically, and reconstruct the event as a 3D scene — semantically labeled and simulation-ready. The edge case that just happened becomes a training asset before anyone opens a rosbag. Real-world incidents are the most valuable data in robotics. We built the pipeline to stop throwing them away. First look 👇 https://lnkd.in/gZd-M9qB If your team is building VLA or Diffusion Policy models and fighting the data pipeline problem, I'd genuinely love to talk. #PhysicalAI #Robotics #RoboticsML #SimToReal #TrainingData

China's "I, Robot" moment will probably be trained by none other than 𝘺𝘰𝘶. ByteDance just released GR-3, a vision-language-action model that brings us closer to truly versatile household robots. The robotics field has long struggled with a fundamental challenge: how do you build robots that can generalize beyond their training data? Most current systems excel at specific tasks but fail when encountering new objects or instructions. This brittleness has kept robots confined to controlled environments rather than our messy, unpredictable homes. GR-3 approaches this through a clever multi-faceted training approach. First, it co-trains on both robot trajectories and massive web-scale vision-language data, allowing it to understand abstract concepts like "put the largest object in the box" - instructions it never saw during robot training. Second, it leverages human trajectory data collected via VR devices, enabling rapid adaptation to new scenarios with just 10 demonstrations per object. The key insight: by combining these diverse data sources with flow-matching for action prediction, the model learns robust representations that transfer across different contexts. GR-3 outperforms the previous state-of-the-art on tasks ranging from generalizable pick-and-place to long-horizon table cleaning and dexterous cloth manipulation. It even shows "true" semantic understanding - distinguishing between similar objects and following complex spatial instructions that would trip up most current systems. In order to push toward robots that can genuinely assist in daily life, approaches like GR-3 that prioritize generalization and efficient adaptation will be crucial. The question now isn't if we'll have capable home robots, but how quickly we can scale these capabilities to handle the full complexity of human environments. And, of course, if we, 𝘢𝘴 𝘢 𝘴𝘰𝘤𝘪𝘦𝘵𝘺, want that in the first place. ↓ 𝐖𝐚𝐧𝐭 𝐭𝐨 𝐤𝐞𝐞𝐩 𝐮𝐩? Join my newsletter with 50k+ readers and be the first to learn about the latest AI research: llmwatch.com 💡