Applications of Temporal Video Dynamics in Robotics

How AI and LLMs Are Teaching Robots to Think in Motion Imagine robots in your facility learning not through costly, action-labeled data but by observing the world through videos, effortlessly adapting to tasks with human, like proficiency. This groundbreaking idea, presented by the Authors, may redefine robotic training by leveraging AI and Large Language Models (LLMs). 🔹 Research Focus The paper explores a pivotal question: can we harness abundant video data to teach robots effectively? Introducing Moto, a framework that converts video frame changes into latent motion tokens, it creates a universal "language" of motion. These tokens enable robots to understand and execute tasks with unprecedented efficiency. 🔹 Latent Motion Tokens At the core of Moto lies the Latent Motion Tokenizer, a model that analyzes transitions between video frames to generate motion tokens. These tokens abstract movement patterns in a hardware-agnostic manner, enabling robots to learn from any video source. This eliminates the reliance on expensive action-labeled datasets and fosters more scalable training. 🔹 Pre-training Motion Priors Moto-GPT, the generative model in the framework, leverages these motion tokens through autoregressive pre-training. By predicting future motion tokens based on past sequences, the model develops a deep understanding of motion dynamics, akin to how humans learn language structures. This equips robots with a robust motion knowledge base, allowing them to anticipate and evaluate actions effectively. 🔹 Co-fine-tuning for Precision To bridge the gap between general motion understanding and specific task execution, Moto employs a co-fine-tuning strategy. This process integrates motion priors with smaller sets of action-labeled data, refining the model for real-world applications. Benchmarks such as SIMPLER and CALVIN highlight Moto's exceptional performance, particularly in scenarios with limited labeled data. 🔹 Experiments and Results Robots trained with Moto outperformed traditional methods, demonstrating faster learning, greater adaptability, and reduced dependency on labeled datasets. By focusing on motion dynamics over static frame details, Moto not only accelerates training but also enhances robot performance across varied environments. 📌 Business Value This innovation signals a paradigm shift. By leveraging Moto’s motion-token-based learning, organizations can drastically cut training costs, increase operational agility, and unlock new possibilities in robotics. Whether in logistics, healthcare, or manufacturing, smarter robots driven by this approach promise smoother workflows and a sharper competitive edge. 👉 How could your organization capitalize on the wealth of video data to train robots? What new opportunities could arise with adaptable, self-learning robotics? 👈 #ArtificialIntelligence #MachineLearning #DeepLearning #Automation #Robotics

Enable **time-awareness** in robots, and entirely new behaviors emerge. Current robotic systems learn a rather "fixed agenda". Regardless of whether they learn from human data or through interactions guided by rewards, robots are trained to complete tasks, but still struggle to adapt to completely unseen scenarios. We believe that one of the fundamental missing pieces is that, unlike humans and animals: - Robot learning systems are still **blind to time**. Time is not just a clock on the wall — it is **a core part of intelligence**. Humans and animals continuously perceive and use time to guide behavior: we hurry when seconds matter, slow down for precision, anticipate others’ actions, and synchronize with teammates. From cognition to motor control, temporal awareness shapes how we move, plan, and collaborate. To close 2025, we’re excited to share our new paper: Time as a Control Dimension in Robot Learning. In this work, we introduce a framework that enables robots to **explicitly perceive and reason about time as a first-class variable**. We achieve this by changing the speed of the internal time lapse of robots. You get many desirable emergent capabilities from a single learned policy:    ⏱️ accelerate when time is scarce 🛠️ slow down for careful, precise manipulation 🤝 synchronize with humans or other robots 🧭 recover from disturbances and still finish on time 🔇 operate more quietly and safely in the real world Across stacking, granular pouring, drawer opening, multi-agent delivery, and human-in-the-loop control, time-aware policies show faster execution, higher robustness, better sim-to-real transfer, and intuitive human controllability — without retraining. 📄 Paper: https://lnkd.in/eMJesfcQ 💻 Code: https://lnkd.in/eZz2eSnb 🎥 Video: https://lnkd.in/ecvm2WYk Awesome work led by our PhD student, Yinsen Jia, at General Robotics Lab. A great way to end 2025. Stay tuned to even more discovery in 2026 🚀

Robotics data is expensive and slow to collect. A lot of videos are available online, but not readily usable by robotics because of lack of action labels. AMPLIFY solves this problem by learning Actionless Motion Priors that unlock better sample efficiency, generalization, and scaling for robot learning. Our key insight is to factor the problem into two stages: The "what": Predict the visual dynamics required to accomplish a task The "how": Map predicted motions to low-level actions This decoupling enables remarkable generalizability: our policy can perform tasks where we have NO action data, only videos. We outperform SOTA BC baselines on this by 27x 🤯 AMPLIFY is composed of three stages: 1. Motion Tokenization: We track dense keypoint grids through videos and compress their trajectories into discrete motion tokens. 2. Forward Dynamics: Given an image and task description (e.g., "open the box"), we autoregressively predict a sequence of motion tokens representing how keypoints should move over the next second or so. This model can train on ANY text-labeled video data - robot demonstrations, human videos, YouTube videos. 3. Inverse Dynamics: We decode predicted motion tokens into robot actions. This module learns the robot-specific mapping from desired motions to actions. This part can train on ANY robot interaction data - not just expert demonstrations (think off-task data, play data, or even random actions). So, does it actually work? Few-shot learning: Given just 2 action-annotated demos per task, AMPLIFY nearly doubles SOTA few-shot performance on LIBERO. This is possible because our Actionless Motion Priors provide a strong inductive bias that dramatically reduces the amount of robot data needed to train a policy. Cross-embodiment learning: We train the forward dynamics model on both human and robot videos, but the inverse model sees only robot actions. Result: 1.4× average improvement on real-world tasks. Our system successfully transfers motion information from human demonstrations to robot execution. And now my favorite result: AMPLIFY enables zero-shot task generalization. We train on LIBERO-90 tasks and evaluate on tasks where we’ve seen no actions, only pixels. While our best baseline achieves ~2% success, AMPLIFY reaches a 60% average success rate, outperforming SOTA behavior cloning baselines by 27x. This is a new way to train VLAs for robotics which dont always start with large scale teleoperation. Instead of collecting millions of robot demonstrations, we just need to teach robots how to read the language of motion. Then, every video becomes training data. led by Jeremy Collins & Loránd Cheng in collaboration with Kunal Aneja, Albert Wilcox, Benjamin Joffe at College of Computing at Georgia Tech Check out our paper and project page for more details: 📄 Paper: https://lnkd.in/eZif-mB7 🌐 Website: https://lnkd.in/ezXhzWGQ

From dexterous hands to imitation learning from internet videos, Lerrel Pinto’s group keeps dropping breakthroughs that set the tone for robotics. 🧵 At NYU Courant, Lerrel Pinto’s lab has quietly reshaped how robots learn. The old way: an engineer spends 100 hours labeling images so a robot can grasp a single key. Pinto’s way: let a robot arm attempt 50,000 grasps overnight. By correlating what it sees with what succeeds, it teaches itself to grasp almost anything. This is self-supervision. No human labels. https://lnkd.in/dGZEv5um But what if you don’t have 50,000 tries? 📍 DynaMo learns from temporal video dynamics rather than isolated frames. With only 6 demos, a multi-fingered hand mastered complex slide-pick-up tasks and outperformed prior methods across 6 simulated and real environments. https://lnkd.in/deu43gQ8 + https://lnkd.in/dFj_jFYH Next, the lab tackled teleoperation. Most tools were expensive, proprietary, or locked to specific robots. 📍Open Teach is open-source, calibration-free, and works across arms, hands, and mobile manipulators... all for about $500 with a Quest 3. https://lnkd.in/dpEjt_YB But vision alone has limits. Robots need to feel. 📍eFlesh: a 3D-printable tactile sensor, built with $5 of materials and a hobbyist printer. http://e-flesh.com 📍T-Dex: with just 2.5 hours of contact-rich “play data,” robots learned to unstack bowls and open bottles (1.7x better than vision-only). https://lnkd.in/dS_rn9CA 📍Feel The Force (FTF): a glove with sensors lets robots learn policies that predict both motion and tactile response... even picking up a raw egg without breaking it. https://lnkd.in/dzWyhHx5 Then came a bigger leap: remove robot data entirely. 📍EgoZero trained policies only from egocentric human videos (FPV). With ~20 minutes of data, robots achieved 70% zero-shot success on 7 manipulation tasks. https://lnkd.in/dD_aUcg2 From there: can robots learn from the internet? Pinto’s lab is training on large YouTube datasets, showing human video can transfer to physical robot skills. And finally, the real-world test. 📍Robot Utility Models (RUMs) achieved 90% success across 25+ unseen homes... without retraining. https://lnkd.in/dNH_HCat The pattern is clear: the bottleneck isn’t hardware. It’s data. From self-supervised grasping to internet-scale imitation, Pinto’s work reframes robotics as a data problem... just like modern AI. His path: IIT Guwahati → CMU Robotics Institute → UC Berkeley → now Assistant Professor at NYU Courant, part of the CILVR group. He’s not just publishing papers. He’s open-sourcing datasets, tools, and methods; creating the foundation for large-scale robot learning. 🎙️ I spoke with Lerrel Pinto about this shift and his vision for the field: https://lnkd.in/dbGipZsj (All paper links will also be listed in the comments for easy access.)