Overcoming Data Bottlenecks in Robotics Projects

Exciting updates on Project GR00T! We discover a systematic way to scale up robot data, tackling the most painful pain point in robotics. The idea is simple: human collects demonstration on a real robot, and we multiply that data 1000x or more in simulation. Let’s break it down: 1. We use Apple Vision Pro (yes!!) to give the human operator first person control of the humanoid. Vision Pro parses human hand pose and retargets the motion to the robot hand, all in real time. From the human’s point of view, they are immersed in another body like the Avatar. Teleoperation is slow and time-consuming, but we can afford to collect a small amount of data.  2. We use RoboCasa, a generative simulation framework, to multiply the demonstration data by varying the visual appearance and layout of the environment. In Jensen’s keynote video below, the humanoid is now placing the cup in hundreds of kitchens with a huge diversity of textures, furniture, and object placement. We only have 1 physical kitchen at the GEAR Lab in NVIDIA HQ, but we can conjure up infinite ones in simulation. 3. Finally, we apply MimicGen, a technique to multiply the above data even more by varying the *motion* of the robot. MimicGen generates vast number of new action trajectories based on the original human data, and filters out failed ones (e.g. those that drop the cup) to form a much larger dataset. To sum up, given 1 human trajectory with Vision Pro  -> RoboCasa produces N (varying visuals)  -> MimicGen further augments to NxM (varying motions). This is the way to trade compute for expensive human data by GPU-accelerated simulation. A while ago, I mentioned that teleoperation is fundamentally not scalable, because we are always limited by 24 hrs/robot/day in the world of atoms. Our new GR00T synthetic data pipeline breaks this barrier in the world of bits. Scaling has been so much fun for LLMs, and it's finally our turn to have fun in robotics! We are creating tools to enable everyone in the ecosystem to scale up with us: - RoboCasa: our generative simulation framework (Yuke Zhu). It's fully open-source! Here you go: http://robocasa.ai - MimicGen: our generative action framework (Ajay Mandlekar). The code is open-source for robot arms, but we will have another version for humanoid and 5-finger hands: https://lnkd.in/gsRArQXy - We are building a state-of-the-art Apple Vision Pro -> humanoid robot "Avatar" stack. Xiaolong Wang group’s open-source libraries laid the foundation: https://lnkd.in/gUYye7yt - Watch Jensen's keynote yesterday. He cannot hide his excitement about Project GR00T and robot foundation models! https://lnkd.in/g3hZteCG Finally, GEAR lab is hiring! We want the best roboticists in the world to join us on this moon-landing mission to solve physical AGI: https://lnkd.in/gTancpNK

Robotics data is expensive and slow to collect. A lot of videos are available online, but not readily usable by robotics because of lack of action labels. AMPLIFY solves this problem by learning Actionless Motion Priors that unlock better sample efficiency, generalization, and scaling for robot learning. Our key insight is to factor the problem into two stages: The "what": Predict the visual dynamics required to accomplish a task The "how": Map predicted motions to low-level actions This decoupling enables remarkable generalizability: our policy can perform tasks where we have NO action data, only videos. We outperform SOTA BC baselines on this by 27x 🤯 AMPLIFY is composed of three stages: 1. Motion Tokenization: We track dense keypoint grids through videos and compress their trajectories into discrete motion tokens. 2. Forward Dynamics: Given an image and task description (e.g., "open the box"), we autoregressively predict a sequence of motion tokens representing how keypoints should move over the next second or so. This model can train on ANY text-labeled video data - robot demonstrations, human videos, YouTube videos. 3. Inverse Dynamics: We decode predicted motion tokens into robot actions. This module learns the robot-specific mapping from desired motions to actions. This part can train on ANY robot interaction data - not just expert demonstrations (think off-task data, play data, or even random actions). So, does it actually work? Few-shot learning: Given just 2 action-annotated demos per task, AMPLIFY nearly doubles SOTA few-shot performance on LIBERO. This is possible because our Actionless Motion Priors provide a strong inductive bias that dramatically reduces the amount of robot data needed to train a policy. Cross-embodiment learning: We train the forward dynamics model on both human and robot videos, but the inverse model sees only robot actions. Result: 1.4× average improvement on real-world tasks. Our system successfully transfers motion information from human demonstrations to robot execution. And now my favorite result: AMPLIFY enables zero-shot task generalization. We train on LIBERO-90 tasks and evaluate on tasks where we’ve seen no actions, only pixels. While our best baseline achieves ~2% success, AMPLIFY reaches a 60% average success rate, outperforming SOTA behavior cloning baselines by 27x. This is a new way to train VLAs for robotics which dont always start with large scale teleoperation. Instead of collecting millions of robot demonstrations, we just need to teach robots how to read the language of motion. Then, every video becomes training data. led by Jeremy Collins & Loránd Cheng in collaboration with Kunal Aneja, Albert Wilcox, Benjamin Joffe at College of Computing at Georgia Tech Check out our paper and project page for more details: 📄 Paper: https://lnkd.in/eZif-mB7 🌐 Website: https://lnkd.in/ezXhzWGQ

One of the biggest constraints in robot learning is data. Rhoda AI released work on their Direct Video-Action (DVA) Model that breaks through this bottleneck. Instead of predicting robot actions directly, the model imagines what the future should look like as video, then translates that into motion, basically treating robot control as real-time video gen. To do this, Rhoda pre-trains a causal video model on internet-scale video to learn physics, object behavior, and material properties. Today's VLAs do most of this learning through robot demonstrations, which are expensive to teach via teleoperation. From there, Rhoda post-trains on a thin layer of task-specific data. For example, just 11 hours of robot data for a complex, bimanual decanting task. Most VLAs operate on only a few frames of context, which makes long-horizon tasks difficult without hand-engineered scaffolding. The DVA retains hundreds of frames of visual history, so it can track where it is in a multi-step task end-to-end. Cool to see competing ideas in robot learning, and excited for what this means with regards to data efficiency and production reliability for end-to-end models.

I was listening to one of my favorite podcasts last week, Unsupervised Learning by Redpoint Ventures. They had Karol Hausman and Danny Driess (Research Scientist) from Physical Intelligence. Around the 33 minute mark of the podcast they mentioned the need for a tool or infrastructure to help them understand what is in their dataset, particularly given the massive amount of multimodal, time-series data that robotics generates. They outlined what they'd want in such a tool: - Decide what data to collect - Build machinery around understanding the collected data - Understand the quality of the data collected so far - Perform quality assurance at scale - Execute language annotations correctly at scale - Determine how much more data is needed for the model - Identify the optimal strategy for data collection - Provide a bird's-eye view understanding of the entire dataset I was excited by that, cuz, well, I work at FiftyOne and we have a tool that does just that... For understanding what's in your dataset, FiftyOne lets you visually explore massive datasets interactively. When they talked about needing a "bird's-eye view," that's literally what our embedding visualizations provide - you can see your entire dataset in embedding space, revealing clusters, gaps, and outliers. The QA at scale problem? FiftyOne has built-in queries to find labeling mistakes and inconsistent patterns across millions of samples. And for data collection strategy, it shows where your dataset has gaps and where models struggle - no more training for weeks to "get a signal." So I went to Physical Intelligence's Hugging Face org and found their "aloha_pen_uncap" dataset. I parsed it into FiftyOne format to see how well our tool would work with their data. In the process, I implemented a data loader for LeRobot format datasets, which means the entire robotics community can now load their datasets in FiftyOne and get all these benefits. The loader handles the multimodal nature of robotics data, parsing camera views, robot states, and actions. What became clear when I loaded their dataset: - You can visually browse task executions and see patterns in successful vs failed attempts - Embedding visualizations shows clusters of similar robot behaviors - Quality issues like poor lighting or occlusions become immediately apparent It's all open source, and all you need to do to get started is `pip install fiftyone` to see what your data looks like in FiftyOne. The tool mentioned in the podcast already exists, and it's open source!

7 lessons from AirSim: I ran the autonomous systems and robotics research effort at Microsoft for nearly a decade and here are my biggest learnings. Complete blog: https://sca.fo/AAeoC 1. The “PyTorch moment” for robotics needs to come before the “ChatGPT moment”. While there is anticipation towards Foundation Models for robots, scarcity of technical folks well versed in both deep ML and robotics, and a lack of resources for rapid iterations present significant barriers. We need more experts to work on robot and physical intelligence. 2. Most AI workloads on robots can primarily be solved by deep learning. Building robot intelligence requires simultaneously solving a multitude of AI problems, such as perception, state estimation, mapping, planning, control, etc. We are increasingly seeing successes of deep ML across the entire robotics stack. 3. Existing robotic tools are suboptimal for deep ML. Most of the tools originated before the advent of deep ML and cloud and were not designed to address AI. Legacy tools are hard to parallelize on GPU clusters. Infrastructure that is data first, parallelizable, and integrates cloud deeply throughout the robot’s lifecycle is a must. 4. Robotic foundation mosaics + agentic architectures are more likely to deliver than monolithic robot foundation models. The ability to program robots efficiently is one of the most requested use cases and a research area in itself. It currently takes a technical team weeks to program robot behavior. It is clear that foundation mosaics and agentic architecture can deliver huge value now. 5. Cloud + connectivity trumps compute on edge – Yes, even for robotics! Most operator-based robot enterprises either discard or minimally catalog the data due to a lack of data management pipelines and connectivity. Given that robotics is truly a multitasking domain – a robot needs to solve for multiple tasks at once. Connection to the cloud for data management, model refinement, and the ability to make several inference calls simultaneously would be a game changer. 6. Current approaches to robot AI Safety are inadequate Safety research for robotics is at an interesting crossroads. Neurosymbolic representation and analysis is likely an important technique that will enable the application of safety frameworks to robotics. 7. Open source can add to the overhead As a strong advocate for open-source, much of my work has been shared. While open-source offers many benefits, there are a few challenges, especially for robotics, that are less frequently discussed: Robotics is a fragmented and siloed field, and likely initially there will be more users than contributors. Within large orgs, the scope of open-source initiatives may also face limits. AirSim pushed the boundaries of the technology and provided a deep insight into R&D processes. The future of robotics will be built on the principle of being open. Stay tuned as we continue to build @Scafoai

If you’ve spent time building digital twins in NVIDIA Isaac Sim / NVIDIA Omniverse, you already know the real bottleneck isn’t simulation, it’s data creation. The usual pipeline is heavy. You spend days setting up assets, materials, scenes, sensors… just to get to a point where you can actually start training. It works, but it’s slow. Lately, I’ve been experimenting with Gaussian splat–based reconstruction (tools like World Labs), and it genuinely feels like a shift in how we approach digital twins. Instead of building everything from scratch, you just capture a short video of a real environment and reconstruct it into a dense 3D radiance field. From there, you can render consistent multi-view data, try different camera setups, and create new scenarios pretty quickly. Now, these twins aren’t perfectly 1:1 in terms of physics or accuracy — but that’s not really the point. What they do give you is a very natural, photorealistic view of the real world in minutes. And that alone speeds things up a lot. You’re no longer blocked by environment creation, which means faster iteration, faster experimentation, and honestly, a much smoother development cycle. Beyond just training, this opens up some interesting directions: Digital twins for monitoring and simulation Copilots that can actually reason over real environments Tighter sim-to-real workflows without spending weeks building scenes I’ll be sharing a few example scenes to showcase the tech (not production work, just to give a sense of what’s possible). Feels like we’re in a really exciting phase for robotics and digital twins right now where creating usable, realistic worlds is becoming the easy part! #nvidia #Isaacsim #isaaclab #digitaltwin #ros2 #simulation #groot #copilot #vla #robotics #robot #monitoring #training #datageneration #usd #gaussian #worldgeneration #omniverse

🚀 RoboCade: Gamifying Robot Data Collection is out on arXiv — and I’m thrilled to share this collaborative work with the community! One of the biggest bottlenecks in robotics today is scaling human demonstration data for imitation learning. Traditional collection is costly, tedious, and limited to experts with access to hardware. So we asked: 👉 Can we make robot data collection accessible, engaging, and scalable — even for non-experts? That’s where RoboCade comes in: 🎮 A gamified remote teleoperation platform that transforms robot demo collection into an interactive game-like experience. 👥 Designed to engage general users — with visual feedback, progress bars, badges, leaderboards, and more — while still generating useful data for downstream policy training. Key results: ✔️ Remote players collected data that, when co-trained with traditional demos, boosted policy success on real tasks (+16 – 56%). ✔️ In user studies, beginners found RoboCade significantly more enjoyable and motivating than standard interfaces (+24%). ✔️ We also propose principles for gamified task design so the collected data actually helps with real manipulation challenges. Why this matters: 🔹 Broadening participation in robotics research beyond labs and experts 🔹 Intrinsic motivation rather than paying for data labeling 🔹 A scalable crowd-sourced pipeline for future robot learning systems Huge thanks to Suvir Mirchandani, Mia Tang, Jubayer Ibn Hamid, Michael Cho, and Dorsa Sadigh for the collaboration. 🔧🤝 Read the full paper on arXiv — and check out our demo videos at https://lnkd.in/gjyE6A5S #Robotics #ImitationLearning #HumanAI #Crowdsourcing #Gamification #MachineLearning

The robotics community has a name for it now: the 100,000-year data gap. You can't scrape robot training data the way you scrape text. It has to be built. And the two options most teams have — teleoperation and hand-authored simulation — are either too expensive to scale or too synthetic to trust at deployment. Here's the part that kept me up at night: Every time a robot hesitates, clips something, or triggers a safety stop in the real world, that's ground-truth data. It's the exact edge case your sim never generated. It has trajectory, context, spatial geometry, failure signature. And in the current workflow, it gets reset and discarded. The failure repeats. The training set stays thin. The sim-to-real gap stays wide. We built Reconstructiv to close that loop. When an incident happens on a real fleet, we detect it, capture the logs and video automatically, and reconstruct the event as a 3D scene — semantically labeled and simulation-ready. The edge case that just happened becomes a training asset before anyone opens a rosbag. Real-world incidents are the most valuable data in robotics. We built the pipeline to stop throwing them away. First look 👇 https://lnkd.in/gZd-M9qB If your team is building VLA or Diffusion Policy models and fighting the data pipeline problem, I'd genuinely love to talk. #PhysicalAI #Robotics #RoboticsML #SimToReal #TrainingData

One of the core challenges in robotics is making robots generalize across tasks, environments, sensors, and data sources. Traditionally, robot learning has been siloed: one model per task, per dataset, per modality. This leads to expensive retraining and poor adaptability. A recent paper from MIT CSAIL introduces PoCo (Policy Composition from and for Heterogeneous Robot Learning), a diffusion-based framework that directly tackles this bottleneck. 🔹The idea: Instead of training a single “one-size-fits-all” model, PoCo allows us to train multiple policies (e.g., task-specific, domain-specific, behavior-constrained) separately, then compose them at inference time. The policies are built as diffusion models over action trajectories, which makes their combination mathematically flexible. This could be a foundational step towards true generalist robots, capable of adapting to diverse and unpredictable environments. Full paper: https://lnkd.in/eim6Se4r