Improving Robot Performance Using Real-World Data

We are excited to share our latest work, "Learning on the Fly: Rapid Policy Adaptation via Differentiable Simulation", where a policy learns to adapt in the real world to unknown disturbances within 5 seconds, both with and without explicit state estimation, directly from visual features. Code released! PDF: https://lnkd.in/eZpWW7dS Project Page: https://lnkd.in/e5bnnipt Starting from a simple analytical dynamics model, the system continuously learns residual dynamics from real-world data and embeds the refined model into a differentiable simulator. This enables fast, gradient-based policy updates that are far more sample-efficient than classical #ReinforcementLearning. We demonstrate rapid adaptation in <5 seconds in agile quadrotor control under challenging conditions, including added payloads, wind disturbances, and large sim-to-real gaps. In real-world experiments, our method reduces hovering error by up to 81% compared to L1-MPC and 55% compared to PPO-based adaptive methods. It also operates directly from visual features without explicit state estimation. Reference:  “Learning on the Fly: Rapid Policy Adaptation via Differentiable Simulation”  IEEE Robotics and Automation Letters, 2026 PDF: https://lnkd.in/eZpWW7dS Video: https://lnkd.in/eSHeKdkr  Code: https://lnkd.in/edidHJng Website: https://lnkd.in/e5bnnipt Kudos to Michael Pan, Jiaxu Xing, Rudolf Reiter, Daniel(Yifan) Zhai, Elie Aljalbout! UZH Department of Informatics, UZH.ai, University of Zurich, UZH Innovation Hub, European Research Council (ERC), AUTOASSESS

Robot models get better only when humans feed them more demos. This one improves by learning from its own mistakes. pi*0.6 is a new VLA from Physical Intelligence, that can refine its skills through real-world RL, not just teleop data. The team calls the method Recap, and from what I can see, the gains are not small. A quick summary: ✅ Learns from its own rollouts using a value function trained across all data ✅ Humans only step in when the robot is about to drift too far ✅ Every correction updates the model and improves future rollouts ✅ Works across real tasks like espresso prep, laundry, and box assembly ✅ Throughput more than doubles on hard tasks, with far fewer failure cases What stands out is the structure: a general policy, a shared value function, and a loop where the robot collects data, improves the critic, then improves itself again. No huge fleets of teleoperators. No massive manual resets. If VLAs can reliably self-improve in the real world, the bottleneck shifts. Data becomes cheaper. Deployment becomes the real test bench. Full paper, videos, and method details here: https://lnkd.in/dgCeZdjT

After building general base models, real-world RL is the endgame. Robots need to be able to quickly adapt to new situations and fix their mistakes on the fly. A base model that can pick up a screwdriver is great, but it's only valuable in production if it can consistently align with a tiny screw at submillimeter precision. Today's models can't do that. Physical Intelligence introduced RL Tokens (RLT), a method that lets a small RL policy sit on top of their base VLA model and refine just the precise, critical phase of a task. No need to fine-tune; instead, the robot can learn from hours (or even minutes) of real-world practice directly on board. The results showed that the RL policy actually executed faster than human teleoperation on half the trials. Across all four tasks they tested, RLT sped up the hardest phases by up to 3x. This is exciting because it provides a pathway for foundation models to achieve production-grade reliability. A robot that can learn in real time can adapt to dynamic conditions in the real world. Interested to see who's first to ship something like this in a real production line.

Just an ordinary day at a robotics company. Progress looks like chaos. We watch the viral backflips and perfect precision. We rarely see the thousands of slips, collisions, and face-plants that happen on the lab floor to get there. This isn't clumsy engineering; it's the "Sim-to-Real" gap in action. The difference between code and concrete is the most valuable data a robotics company possesses: ✔️ Reinforcement Learning (The Grind): In a simulation, a robot can train for 1,000 years in a single day. But real-world physics is unforgiving. Every one of these falls is a high-fidelity data point that refines the neural network's balance policy. ✔️ Resilience over Perfection: The goal isn't to build a robot that never falls. It's to build a system that can recover from failure in milliseconds, autonomously, without human intervention. ✔️ Domain Randomization: You see chaos; the algorithm sees variety. Kicking the robot, slippery floors, and random obstacles are features, not bugs. They force the model to generalize beyond its training set.

Video world models look good, but often don't follow the basic laws of physics and 3D geometry. This research achieved a 64% reduction in navigation error by teaching AI to prioritize physical reality, here's how they did it: Most world models only predict pixels for visual realism, leading to wobbling depths and drifting paths that make simulations useless for real-world robots. To fix this, the team developed GrndCtrl, which treats world modeling as a verifiable reasoning task. Instead of just trying to look right, the model generates multiple potential video futures and puts them through a physical audition. Specialized "judges" grade these videos based on 3D math, checking if the rotation, movement, and depth actually add up. By using an optimization method called GRPO, the model learns to favor the versions that obey the laws of physics over the ones that just look pretty. The result is a breakthrough in actionable output for robots. By rewarding structural consistency, the system achieved a 64% reduction in translation error on complex, unseen paths. It can now simulate a world it can actually inhabit. Project Page: https://lnkd.in/gKydQbvz Paper: https://lnkd.in/gM9UVcJE #Robotics #WorldModels #ComputerVision

World model trained on 44,000 hours of human videos! 🌍 NVIDIA Robotics, UC Berkeley, HKUST, and UT Austin just released DreamDojo, a foundation world model for robots trained on the largest video dataset to date for world model pretraining. The closer it gets to GTC, the more NVIDIA is cooking. 😮💨 The dataset: 44,000 hours of diverse human egocentric videos. That's 15x longer duration, 96x more skills, and 2,000x more scenes than the previously largest dataset for world model training. DreamDojo learns comprehensive physical knowledge from large-scale human data through pre-training with latent actions, then post-trains on specific robot embodiments with continuous robot actions. Strong generalization to diverse objects and environments after post-training. The model produces realistic action-conditioned rollouts for GR-1, G1, AgiBot, and YAM robots across wide-ranging environments and object interactions. After distillation, the model achieves long-horizon autoregressive generation with stable real-time interactions at 10 FPS for over 1 minute. The distillation pipeline enables deployment speed comparable to direct policy execution. Live teleoperation with real-time rollout generation, reliable policy evaluation without real-world deployment, and model-based planning for test-time improvement. Comparison with baselines shows DreamDojo generates more accurate physical interactions due to large-scale human data pretraining. The model learned physics and object manipulation priors from massive human video data, then transferred this knowledge to robot control. This is the same pattern as language models: pre-train on massive human-generated data (text → video), then fine-tune for specific tasks (completion → robot control). Congrats Jim Fan and team behind this project: https://lnkd.in/dV-nzDip ~~ ♻️ Join the weekly robotics newsletter, and never miss any news → ziegler.substack.com