How Simulation Data Impacts Robotics Performance

Exciting updates on Project GR00T! We discover a systematic way to scale up robot data, tackling the most painful pain point in robotics. The idea is simple: human collects demonstration on a real robot, and we multiply that data 1000x or more in simulation. Let’s break it down: 1. We use Apple Vision Pro (yes!!) to give the human operator first person control of the humanoid. Vision Pro parses human hand pose and retargets the motion to the robot hand, all in real time. From the human’s point of view, they are immersed in another body like the Avatar. Teleoperation is slow and time-consuming, but we can afford to collect a small amount of data.  2. We use RoboCasa, a generative simulation framework, to multiply the demonstration data by varying the visual appearance and layout of the environment. In Jensen’s keynote video below, the humanoid is now placing the cup in hundreds of kitchens with a huge diversity of textures, furniture, and object placement. We only have 1 physical kitchen at the GEAR Lab in NVIDIA HQ, but we can conjure up infinite ones in simulation. 3. Finally, we apply MimicGen, a technique to multiply the above data even more by varying the *motion* of the robot. MimicGen generates vast number of new action trajectories based on the original human data, and filters out failed ones (e.g. those that drop the cup) to form a much larger dataset. To sum up, given 1 human trajectory with Vision Pro  -> RoboCasa produces N (varying visuals)  -> MimicGen further augments to NxM (varying motions). This is the way to trade compute for expensive human data by GPU-accelerated simulation. A while ago, I mentioned that teleoperation is fundamentally not scalable, because we are always limited by 24 hrs/robot/day in the world of atoms. Our new GR00T synthetic data pipeline breaks this barrier in the world of bits. Scaling has been so much fun for LLMs, and it's finally our turn to have fun in robotics! We are creating tools to enable everyone in the ecosystem to scale up with us: - RoboCasa: our generative simulation framework (Yuke Zhu). It's fully open-source! Here you go: http://robocasa.ai - MimicGen: our generative action framework (Ajay Mandlekar). The code is open-source for robot arms, but we will have another version for humanoid and 5-finger hands: https://lnkd.in/gsRArQXy - We are building a state-of-the-art Apple Vision Pro -> humanoid robot "Avatar" stack. Xiaolong Wang group’s open-source libraries laid the foundation: https://lnkd.in/gUYye7yt - Watch Jensen's keynote yesterday. He cannot hide his excitement about Project GR00T and robot foundation models! https://lnkd.in/g3hZteCG Finally, GEAR lab is hiring! We want the best roboticists in the world to join us on this moon-landing mission to solve physical AGI: https://lnkd.in/gTancpNK

You cannot train physical AI on reality alone. There is not enough of it. Jensen Huang explains why NVIDIA built Alpamayo, a robotics model that learns from synthetic data grounded in physics. The problem is fundamental. Teaching physical AI like autonomous vehicles or robotics requires vast amounts of diverse interaction data. Videos exist. Lots of videos. But hardly enough to capture the diversity and type of interactions needed. So NVIDIA transformed compute into data. Using synthetic data generation grounded and conditioned by laws of physics, they can selectively generate training scenarios reality cannot provide. The example Huang shows is remarkable. A basic traffic simulator output gets fed into Cosmos AI world model. What emerges is physically based, physically plausible surround video that AI can learn from. This solves a constraint that limited physical AI development. You cannot train autonomous systems on every possible scenario by recording reality. There are not enough cameras, time, or situations. But you can simulate physics accurately enough that AI trained on synthetic data generalizes to real environments. Why this matters beyond autonomous vehicles. Any AI learning physical interactions faces the same data scarcity problem. Manufacturing robots, warehouse automation, infrastructure inspection, medical robotics. All require training on scenarios that are rare, dangerous, or impossible to capture at scale. Synthetic data generation grounded in physics laws becomes essential infrastructure for physical AI deployment. The organizations building AI for physical systems will either master synthetic data generation or remain limited by whatever reality they can record. Watch the full presentation to hear Huang explain how Alpamayo generates training data for autonomous vehicles that think like humans. What physical AI application needs synthetic data because reality cannot provide enough examples?

A humanoid robot costs $90K to break once. AI lets you break thousands... and learn from every fall. My background is mechanical engineering, robotics, and integration & test. But this field is moving so fast with AI that reading articles wasn't cutting it anymore. I felt out of the loop, so... I recently upgraded my personal setup to support AI training workloads and ran my first experiment: Teaching a bipedal (two-legged) humanoid robot to navigate a custom parkour course using reinforcement learning in NVIDIA Isaac Lab 5.1. But before I share what I learned, let me explain what's actually happening under the hood. A GPU-accelerated AI agent runs thousands of virtual robots in parallel. Each one learns from its own falls and successes simultaneously. The AI develops a "control policy," which is the brain that tells a robot how to move through the physical world. Why does this matter? Because what once required million-dollar labs and months of physical testing can now run on a single AI-capable GPU in hours. Robotics R&D is becoming software-first. Here's what that looked like for this experiment: 76 minutes of CUDA-accelerated training time. 393 million training steps. 4,096 robots learning in parallel on my RTX 5080. So what did I learn so far? Three things stood out to me: 》The setup before you can hit "Run" is a challenge. It took me seven hours to troubleshoot versioning, packages, and dependencies before I could run anything. I forced myself to do it manually because I wanted to understand what's under the hood. YouTube tutorials hit their limit quickly, but thankfully the NVIDIA developer forums saved me. 》The cost case is undeniable. A Unitree H1 costs around $90K. I *virtually* crashed thousands of them. My damage bill? $0. Simulation lets you fail-forward at scale. This gets you to a solid starting point for physical testing, but... 》The Sim-to-Real gap is real. This policy works well in simulation, but I couldn't get a feel for stress points, sensor behavior, or true stability. Failure is not predictable and happens at the edges. The next step would be to transfer this policy to a physical robot, gather real-world data, and continuously aligning the simulation to close that gap. The key thing here is: Testing real hardware is expensive. Simulation in software is cheap. How can you leverage both, intelligently? The benefit isn't limited to cost savings. This workflow also compresses developmental cycles and allows you to field systems faster. Do you think virtual simulation is a game-changer that is here to stay, or a fad? How would you build confidence in a robotic control policy that is trained in a virtual world? #robotics #ai #nvidia #omniverse #isaaclab ~~~~~~~~ Citations: NVIDIA IsaacLab -> https://lnkd.in/ekVMDnDc RSL-RL -> https://lnkd.in/eJye3XTW Unitree H1-> unitree.com/h1/ Note: this is an educational personal project. Opinions are my own, no affiliation or endorsement.

Is real-world data still the bottleneck for robot learning? We just flipped the script. Zero real-world data. ➔ Autonomous humanoid loco-manipulation in reality. I’m excited to introduce VIRAL: Visual Sim-to-Real at Scale. The robotics community has long relied on expensive, slow, human-collected data. We took a different path. By training entirely inside NVIDIA Isaac Lab, we achieved 54 autonomous cycles (walk, stand, place, pick, turn) in the real world using a simple recipe: RL + Simulation + GPUs. Here is how we achieved photorealistic sim-to-real transfer without a single drop of real-world data: 1. The Pipeline (Teacher ➔ Student) We accelerate physics by 10,000x real-time. We train a privileged teacher with full state access in sim, then distill that into a vision-based student policy using DAgger and Behavior Cloning. 2. Scale is not "Optional" We scaled visual sim-to-real compute up to 64 GPUs. We discovered that for long-horizon tasks like loco-manipulation, large-scale simulation is strictly necessary for convergence and robustness. 3. Bridging the Reality Gap To handle complex hardware (like 3-fingered dexterous hands), we performed rigorous System Identification (SysID). The difference in physics matching was night and day. 4. The "Free Lunch" Sim-to-real is incredibly hard to build (it took us 6 months of infrastructure work). But once solved, you get generalization for free. VIRAL handles diverse spatial arrangements and visual variations without any real-world fine-tuning. Check out the full breakdown:  📄 Paper: https://lnkd.in/eZE6GzEd  🌐 Website: https://lnkd.in/euRajeVm A huge congratulations to the incredible team behind this work: Tairan He*, Zi Wang*, Haoru Xue*, Qingwei Ben*, Zhengyi Luo, Wenli Xiao, Ye Yuan, Xingye Da, Fernando Castañeda, Shankar Sastry, Changliu Liu, Guanya Shi. GEAR Leads: Jim Fan†, Yuke Zhu†

𝐑𝐞𝐢𝐧𝐟𝐨𝐫𝐜𝐞𝐦𝐞𝐧𝐭 𝐋𝐞𝐚𝐫𝐧𝐢𝐧𝐠 𝐈𝐬 𝐍𝐨𝐭 𝐄𝐯𝐨𝐥𝐯𝐢𝐧𝐠 𝐑𝐨𝐛𝐨𝐭𝐢𝐜𝐬. 𝐈𝐭 𝐈𝐬 𝐑𝐞𝐰𝐢𝐫𝐢𝐧𝐠 𝐈𝐭. Reinforcement learning has crossed the line from academic promise into measurable industrial and real world dominance. Robots are no longer executing hand coded instructions. They are learning through consequence, adapting through uncertainty, and improving through reward. This is the moment where automation becomes intelligence. In high fidelity simulation environments, modern RL policies now achieve performance levels that were considered unattainable just a few years ago. In a recent dual arm robotic assembly system, the policy reached a 99.8 percent success rate across 35,000 training episodes. Mean cycle times stabilized at under five seconds while maintaining precision insertion under randomized joint noise. This is not marginal improvement. This is near perfect reliability in a task that historically caused massive failure rates under traditional control. When transferred into the physical world, those same learned behaviors did not collapse. They improved. This is what true autonomy looks like. Not scripted motion. Adaptive force, perception, and decision making in real time. Virtual reality is now accelerating that loop even further. In distributed supervisory control systems that combine immersive VR interfaces with deep reinforcement learning, operators issue high level goals while autonomous policies execute low level motion. In recent trials, this hybrid architecture reduced task completion time by over 50 percent and eliminated collisions entirely. Operator workload dropped significantly while system usability scores exceeded 84 out of 100. Human intent and machine intelligence are no longer competing. They are converging. At scale, reinforcement learning is now coordinating swarms of autonomous systems using graph based policies that distribute decision making across hundreds of agents. Efficiency gains exceeding 90 percent in cooperative tasks such as navigation, sensing, and area coverage are now being reported. At the edge, quantized RL models running on compact hardware are executing real time inference under extreme size, weight, and power constraints. Autonomy is moving out of the lab and into everything. The deeper truth is this: We are no longer programming robots. We are training them. Simulation builds the mind. Real world deployment proves it. Virtual reality sharpens it. Multi agent learning scales it. Reinforcement learning is becoming the nervous system of the next generation of machines. And the results are no longer theoretical. They are measurable, repeatable, and already reshaping what autonomy means. #changetheworld

Video world models look good, but often don't follow the basic laws of physics and 3D geometry. This research achieved a 64% reduction in navigation error by teaching AI to prioritize physical reality, here's how they did it: Most world models only predict pixels for visual realism, leading to wobbling depths and drifting paths that make simulations useless for real-world robots. To fix this, the team developed GrndCtrl, which treats world modeling as a verifiable reasoning task. Instead of just trying to look right, the model generates multiple potential video futures and puts them through a physical audition. Specialized "judges" grade these videos based on 3D math, checking if the rotation, movement, and depth actually add up. By using an optimization method called GRPO, the model learns to favor the versions that obey the laws of physics over the ones that just look pretty. The result is a breakthrough in actionable output for robots. By rewarding structural consistency, the system achieved a 64% reduction in translation error on complex, unseen paths. It can now simulate a world it can actually inhabit. Project Page: https://lnkd.in/gKydQbvz Paper: https://lnkd.in/gM9UVcJE #Robotics #WorldModels #ComputerVision

I had the chance to visit Lightwheel during NVIDIA GTC! 🔥 Most robotics simulations today look good visually. But under the hood, physics is often just an approximation. Friction, contact, deformation, the things that actually determine whether a robot succeeds or fails, are usually simplified. That’s one of the main reasons why: something that works in simulation often breaks in the real world. Lightwheel’s approach is different. With SimReady, the idea is not to guess physics, but to measure it. 📏 You take real-world data (forces, materials, interactions), bring it into simulation, and generate assets that behave the same way digitally as they do physically. The second piece is just as important: evaluation.That’s where RoboFinals comes in. Instead of isolated demos, it provides a structured way to test robots across many environments and diverse sets of tasks. Because right now, “this looks good in a video” is still too often the evaluation method. Lightwheel has joined the Newton Technical Steering Committee along with NVIDIA, Google Deepmind and Disney, to help lead the future of physically grounded simulation. Congrats Steve Xie, Ph.D. and team! NVIDIA Robotics 💚 ~~ ♻️ Join the weekly robotics newsletter, and never miss any news → ziegler.substack.com

Simulation lets us train millions of trajectories. Reality tests whether any of them actually work. The Sim2Real gap isn’t one problem — it’s three: 1️⃣ Visual Shift (Real ≠ Synthetic) Real scenes have noise, clutter, glare, shadows, messy backgrounds. Sim rarely does. 2️⃣ Physics Shift (Approximation ≠ Reality) Small errors in friction, damping, mass, or latency → huge drift in behavior. 3️⃣ Embodiment Shift (Robot ≠ Robot-in-Sim) Morphology, joint limits, actuator dynamics — nothing matches perfectly. What works today? • Domain Randomization — vary textures, lights, physics, noise until the policy generalizes by force • Domain Adaptation — align real + sim feature distributions • System Identification — tune sim from real sensor measurements • Real-to-Sim Feedback Loops — use a tiny amount of real data to anchor the model As robotics foundation models scale, most of their data will come from simulation. Teams who master domain adaptation will be the ones who can actually deploy these models on physical robots — not just in demos. I added my favorite papers, frameworks, and tools in the comments 👇

Robotics data for physical AI is front and center this year. From GTC's heavy focus on data infrastructure to human data ecosystems like EgoVerse, the field is waking up to the bottleneck of scaling robotics data. And there's real divergence in how people think about quantity, quality, modality, and diversity. Aurora Feng, Albert K., and I have been looking at this problem. We put together a robotics data infrastructure market map and an open vendor/collector list for the community. We structured the market map around the robot data workflow end to end: - Simulations & Evaluation: Generating synthetic + real data, benchmarking model performance. - Curation & Labeling: Annotation, QA, slicing the data that actually matters. - Ingestion & Sync: Time-aligning multimodal sensor streams into usable formats. - Storage & Indexing: Making robotics data searchable and retrievable. - Deployment & Ops: Fleet telemetry, incident detection, closing the loop back into the data stack. Most companies span multiple layers. The interesting question is which layers are still wide open. The LLM instinct is more data = better model. In robotics that logic weakens, and the scaling laws don't directly apply (yet). What actually matters when evaluating data: - Outcome quality: Are success/failure trajectories labeled correctly? Failure trajectories can be quite useful if labeled correctly. Some vendors mix them in with successes to inflate volume and price. - Distribution diversity: How varied are the demonstrations? Different initial states, grasp points, camera views. Narrow distributions produce brittle models. - Annotation granularity: Task-level labels aren't enough. You need trajectory-level and step-level annotations. It's not about dataset size. It's about whether the data teaches the right things. Most robot data pipelines are still duct-taped together. Some gaps we see: - Ingestion & sync is still painfully manual. Time-aligning video, depth, force, and actions across sensors is an engineering tax every team pays independently. - Curation & QA tooling was built for AV, not manipulation or dexterous tasks. - Evaluation infra barely exists outside big labs. Most teams can't tell if new data actually improves their models. Ultimately, the most valuable data infra companies will be the platforms that match the right data modalities to the right model architectures and evolve alongside the models themselves. Who are we missing? If you're building data infra or have insights on what data inputs matter most, we want to hear from you. Vendor list + details in the first comment ↓