Physics-Based Simulation for Robotics Data Generation

What if a robot could simulate the physical world from a single image. [📍Bookmark Paper & GitHub for later] PointWorld-1B from Stanford and NVIDIA is a large 3D world model that predicts how an entire scene will move, given RGB-D input and robot actions. The key idea is simple but powerful: Actions are not joint angles❗️ They are 3D point flows sampled from the robot’s own geometry. The model reasons in the same space where physics actually happens. • State and action are unified as 3D point trajectories. • One forward pass predicts full-scene motion for one second. • No object masks, no trackers, no material priors. • Trained on ~500 hours of real and simulated robot interaction data. • Micrometer-level trajectory error, thinner than a human hair. • Works across embodiments, from single arm to bimanual humanoid. The model is then used inside an MPC planner to push objects, manipulate cloth, and use tools, all zero-shot, from a single fixed camera and without finetuning. This feels like a shift from “learning policies” to “learning physics in 3D”. Thanks for sharing, @wenlong_huang 📍Project point-world.github.io 📍Paper arxiv.org/abs/2601.03782 📍GitHub https://lnkd.in/dqsjUTxg (will be published soon) —— Weekly robotics and AI insights. Subscribe free: scalingdeep.tech

Exciting updates on Project GR00T! We discover a systematic way to scale up robot data, tackling the most painful pain point in robotics. The idea is simple: human collects demonstration on a real robot, and we multiply that data 1000x or more in simulation. Let’s break it down: 1. We use Apple Vision Pro (yes!!) to give the human operator first person control of the humanoid. Vision Pro parses human hand pose and retargets the motion to the robot hand, all in real time. From the human’s point of view, they are immersed in another body like the Avatar. Teleoperation is slow and time-consuming, but we can afford to collect a small amount of data.  2. We use RoboCasa, a generative simulation framework, to multiply the demonstration data by varying the visual appearance and layout of the environment. In Jensen’s keynote video below, the humanoid is now placing the cup in hundreds of kitchens with a huge diversity of textures, furniture, and object placement. We only have 1 physical kitchen at the GEAR Lab in NVIDIA HQ, but we can conjure up infinite ones in simulation. 3. Finally, we apply MimicGen, a technique to multiply the above data even more by varying the *motion* of the robot. MimicGen generates vast number of new action trajectories based on the original human data, and filters out failed ones (e.g. those that drop the cup) to form a much larger dataset. To sum up, given 1 human trajectory with Vision Pro  -> RoboCasa produces N (varying visuals)  -> MimicGen further augments to NxM (varying motions). This is the way to trade compute for expensive human data by GPU-accelerated simulation. A while ago, I mentioned that teleoperation is fundamentally not scalable, because we are always limited by 24 hrs/robot/day in the world of atoms. Our new GR00T synthetic data pipeline breaks this barrier in the world of bits. Scaling has been so much fun for LLMs, and it's finally our turn to have fun in robotics! We are creating tools to enable everyone in the ecosystem to scale up with us: - RoboCasa: our generative simulation framework (Yuke Zhu). It's fully open-source! Here you go: http://robocasa.ai - MimicGen: our generative action framework (Ajay Mandlekar). The code is open-source for robot arms, but we will have another version for humanoid and 5-finger hands: https://lnkd.in/gsRArQXy - We are building a state-of-the-art Apple Vision Pro -> humanoid robot "Avatar" stack. Xiaolong Wang group’s open-source libraries laid the foundation: https://lnkd.in/gUYye7yt - Watch Jensen's keynote yesterday. He cannot hide his excitement about Project GR00T and robot foundation models! https://lnkd.in/g3hZteCG Finally, GEAR lab is hiring! We want the best roboticists in the world to join us on this moon-landing mission to solve physical AGI: https://lnkd.in/gTancpNK

You cannot train physical AI on reality alone. There is not enough of it. Jensen Huang explains why NVIDIA built Alpamayo, a robotics model that learns from synthetic data grounded in physics. The problem is fundamental. Teaching physical AI like autonomous vehicles or robotics requires vast amounts of diverse interaction data. Videos exist. Lots of videos. But hardly enough to capture the diversity and type of interactions needed. So NVIDIA transformed compute into data. Using synthetic data generation grounded and conditioned by laws of physics, they can selectively generate training scenarios reality cannot provide. The example Huang shows is remarkable. A basic traffic simulator output gets fed into Cosmos AI world model. What emerges is physically based, physically plausible surround video that AI can learn from. This solves a constraint that limited physical AI development. You cannot train autonomous systems on every possible scenario by recording reality. There are not enough cameras, time, or situations. But you can simulate physics accurately enough that AI trained on synthetic data generalizes to real environments. Why this matters beyond autonomous vehicles. Any AI learning physical interactions faces the same data scarcity problem. Manufacturing robots, warehouse automation, infrastructure inspection, medical robotics. All require training on scenarios that are rare, dangerous, or impossible to capture at scale. Synthetic data generation grounded in physics laws becomes essential infrastructure for physical AI deployment. The organizations building AI for physical systems will either master synthetic data generation or remain limited by whatever reality they can record. Watch the full presentation to hear Huang explain how Alpamayo generates training data for autonomous vehicles that think like humans. What physical AI application needs synthetic data because reality cannot provide enough examples?

The new Newton physics engine is a game-changer for simulation. It’s not just faster — it’s truer to the real world. ✅ Why it matters: • Real robots operate in messy environments — dust, soil, cables, cloth, fluids. • Traditional simulators simplify or ignore those details. • Newton introduces physics intrinsics — friction, deformation, and material interaction — that make simulations behave like real life. ✅ What this means: • More accurate training data for AI and robotics. • More reliable behavior transfer from sim-to-real. • Fewer surprises when your model meets the real world. ✅ Key advantage: • GPU-native architecture runs thousands of environments at once. • Each environment obeys real-world physical laws. • Simulations now capture how dust swirls, fabric folds, or soil compresses — all in real time. ✅ The result: Simulation that doesn’t just look real — it behaves real. That’s the next frontier for robotics, AI, and digital twins. At FS Studio, we’re already integrating these new physics intrinsics into our synthetic data and digital twin workflows — pushing simulations to mirror the real world more faithfully than ever before. Check out Newton for yourself: https://lnkd.in/gaPVyGin

For my PhysicalAI/Robotics/Autonomous Systems folks - Over the past week I’ve been digging into the NVIDIA Cosmos platform, and I’m genuinely excited by what it means for PhysicalAI, robotics, and autonomy. With our newly published NVIDIA Cosmos Cookbook, we now have a practical, scalable path to generate high-fidelity synthetic data for real-world robotics, autonomous vehicles, and sensor-based systems... All without needing millions of hours of real-world data collection. First, what is Cosmos? Cosmos is a platform purpose-built for physical AI, featuring state-of-the-art generative world foundation models (WFMs), guardrails, and an accelerated data processing and curation pipeline for autonomous vehicle (AV), robotics, and AI agent developers. You may be asking what a WFM is - Simply put, a World Foundation Model is a digital replica of the physical world where Physical AI can safely learn and practice. Why this matters? Synthetic data via Cosmos Transfer lets us vary background, lighting, weather, and other environmental parameters, (As seen in the GIF I uploaded), generating realistic video and sensor data “at will.” That means we can create rare or dangerous scenarios (hard to capture in the real world) in simulation: edge-case driving conditions, complex urban terrain, unusual lighting or weather — all of which matter for robust, safe AVs and robots. For anyone working robotics, autonomy, DoD-grade AI systems, or hybrid physical-digital AI pipelines: now is a great time to take a hard look at Cosmos. I see huge potential from UARCs/FFRDCs to Research Laboratories to the FSI community. NVIDIA Blog Here -> https://lnkd.in/e2_RP_mR Cosmos Github repo -> https://lnkd.in/exUjD4T9 Cosmos Hugging Face repo -> https://lnkd.in/eDyMm6Re NVIDIA Cosmos for Developers Homepage -> https://lnkd.in/ehSnkmGc

Is real-world data still the bottleneck for robot learning? We just flipped the script. Zero real-world data. ➔ Autonomous humanoid loco-manipulation in reality. I’m excited to introduce VIRAL: Visual Sim-to-Real at Scale. The robotics community has long relied on expensive, slow, human-collected data. We took a different path. By training entirely inside NVIDIA Isaac Lab, we achieved 54 autonomous cycles (walk, stand, place, pick, turn) in the real world using a simple recipe: RL + Simulation + GPUs. Here is how we achieved photorealistic sim-to-real transfer without a single drop of real-world data: 1. The Pipeline (Teacher ➔ Student) We accelerate physics by 10,000x real-time. We train a privileged teacher with full state access in sim, then distill that into a vision-based student policy using DAgger and Behavior Cloning. 2. Scale is not "Optional" We scaled visual sim-to-real compute up to 64 GPUs. We discovered that for long-horizon tasks like loco-manipulation, large-scale simulation is strictly necessary for convergence and robustness. 3. Bridging the Reality Gap To handle complex hardware (like 3-fingered dexterous hands), we performed rigorous System Identification (SysID). The difference in physics matching was night and day. 4. The "Free Lunch" Sim-to-real is incredibly hard to build (it took us 6 months of infrastructure work). But once solved, you get generalization for free. VIRAL handles diverse spatial arrangements and visual variations without any real-world fine-tuning. Check out the full breakdown:  📄 Paper: https://lnkd.in/eZE6GzEd  🌐 Website: https://lnkd.in/euRajeVm A huge congratulations to the incredible team behind this work: Tairan He*, Zi Wang*, Haoru Xue*, Qingwei Ben*, Zhengyi Luo, Wenli Xiao, Ye Yuan, Xingye Da, Fernando Castañeda, Shankar Sastry, Changliu Liu, Guanya Shi. GEAR Leads: Jim Fan†, Yuke Zhu†

Simulation lets us train millions of trajectories. Reality tests whether any of them actually work. The Sim2Real gap isn’t one problem — it’s three: 1️⃣ Visual Shift (Real ≠ Synthetic) Real scenes have noise, clutter, glare, shadows, messy backgrounds. Sim rarely does. 2️⃣ Physics Shift (Approximation ≠ Reality) Small errors in friction, damping, mass, or latency → huge drift in behavior. 3️⃣ Embodiment Shift (Robot ≠ Robot-in-Sim) Morphology, joint limits, actuator dynamics — nothing matches perfectly. What works today? • Domain Randomization — vary textures, lights, physics, noise until the policy generalizes by force • Domain Adaptation — align real + sim feature distributions • System Identification — tune sim from real sensor measurements • Real-to-Sim Feedback Loops — use a tiny amount of real data to anchor the model As robotics foundation models scale, most of their data will come from simulation. Teams who master domain adaptation will be the ones who can actually deploy these models on physical robots — not just in demos. I added my favorite papers, frameworks, and tools in the comments 👇