Contact-Implicit Control Methods for Robotics

Robotic loco-manipulation requires coordinated control of both contact force and position, yet many visuomotor policies treat them separately. We propose a unified control policy for legged robots that jointly models force and position — learned without force sensors. By simulating diverse interactions, the policy estimates contact forces from past states and compensates via position and velocity adjustments. Such a policy enables a wide range of manipulation behaviors under varying combinations of force and position inputs, including position tracking, force application, force tracking, and compliant robot behaviors. Additionally, we demonstrate that the learned policy enhances trajectory-based imitation learning pipelines by incorporating essential contact information through its force estimation module, achieving approximately ~39.5% higher success rates across four challenging contact-rich manipulation tasks compared to position-control policies. Experiments on both a quadruped and a humanoid demonstrate the robustness and versatility of our method across diverse scenarios.

Presenting FEELTHEFORCE (FTF): a robot learning system that models human tactile behavior to learn force-sensitive manipulation. Using a tactile glove to measure contact forces and a vision-based model to estimate hand pose, they train a closed-loop policy that continuously predicts the forces needed for manipulation. This policy is re-targeted to a Franka Panda robot with tactile gripper sensors using shared visual and action representa- tions. At execution, a PD controller modulates gripper closure to track predicted forces -enabling precise, force-aware control. This approach grounds robust low- level force control in scalable human supervision, achieving a 77% success rate across 5 force-sensitive manipulation tasks. #research: https://lnkd.in/dXxX7Enw #github: https://lnkd.in/dQVuYTDJ #authors: Ademi Adeniji, Zhuoran (Jolia) Chen, Vincent Liu, Venkatesh Pattabiraman, Raunaq Bhirangi, Pieter Abbeel, Lerrel Pinto, Siddhant Haldar New York University, University of California, Berkeley, NYU Shanghai Controlling fine-grained forces during manipulation remains a core challenge in robotics. While robot policies learned from robot-collected data or simulation show promise, they struggle to generalize across the diverse range of real-world interactions. Learning directly from humans offers a scalable solution, enabling demonstrators to perform skills in their natural embodiment and in everyday environments. However, visual demonstrations alone lack the information needed to infer precise contact forces.

Can my robot cook my food, tidy my messy table, rearrange my dresser and do much much more without ANY demos or real-world training data? Introducing ManipGen: A generalist agent for manipulation that can solve long-horizon robotics tasks entirely zero shot, from text input! Key idea: for many manipulation tasks of interest, they can be decomposed into two phases, contact-free reaching (aka motion planning!) and contact-rich local interaction. The latter is hard to learn, and we take a sim2real transfer approach! We define local policies, which operate in a local region around an object of interest. They are uniquely well-suited to generalization (see below!) and sim2real transfer. This is because they are invariant to: 1) Absolute pose 2) Skill orders 3) Environment configurations As an overview, our approach 1) acquires generalist behaviors for local skills at scale using RL 2) distills these behaviors into visuomotor policies using multitask DAgger and 3) deploys local policies in the real-world using VLMs and motion planning. Phase 1: Train state-based, single-object policies to acquire skills such as picking, placing, opening and closing. We train policies using PPO across thousands of objects, designing reward and observation spaces for efficient learning and effective sim2real transfer. Phase 2: We need visuomotor policies to deploy on robots! We distill single-object experts into multi-task policies using online imitation learning (aka DAgger) that observe local visual (wrist cam) input with edge and hole augmentation to match real-world depth noise. To deploy local policies in the real-world, we decompose the task into components (GPT-4o), estimate where to go using Grounded SAM, and motion plan using Neural MP. For control, we use Industreallib from NVIDIA, an excellent library for sim2real transfer! ManipGen can solve long-horizon tasks in the real-world entirely zero-shot generalizing across objects, poses, environments and scene configurations! We outperform SOTA approaches such as SayCan, OpenVLA, LLMTrajGen and VoxPoser across 50 tasks by 36%, 76%, 62% and 60%! ManipGen exhibits exciting capabilities such as performing manipulation in tight spaces and with clutter, entirely zero-shot! From putting items on the shelf, carefully extracting the red pepper from clutter and putting large items in drawers, ManipGen is quite capable. By training local policies at scale on thousands of objects, ManipGen generalizes to some pretty challenging out of distribution objects that don’t look anything like what was in training, such as pliers and the clamps as well as deformable objects such as the wire. This work was done at Carnegie Mellon University Robotics Institute, with co-lead Min Liu, as well as Deepak Pathak and Russ Salakhutdinov and in collaboration with Walter Talbott, Chen Chen, Ph.D., and Jian Zhang from Apple.  Paper, videos and code (coming soon!) at https://lnkd.in/ekjWPXHM

Simultaneous Tactile Estimation and Control of Extrinsic Contact Simultaneously estimates and controls extrinsic contact with tactile feedback for delicate manipulation tasks. Utilises a factor graph-based estimator-controller framework to fuse tactile measurements and estimate contact state. Plans robot motion to collect measurements from various poses while achieving control objectives like minimising intrinsic wrench and tangential force. Achieves contact localisation error under 1 mm and prevents slipping even in slippery environments. Supports multiple contact formations (point, line, patch) and detects transitions between them. Website: https://lnkd.in/ehen7hSS Paper: https://lnkd.in/eg4qCc88 Code: https://lnkd.in/ev2Bv72N Video: https://lnkd.in/eyc25u73

Why do learned predictive models still fail at planning and control when robots touch the ground? In our new paper, we show that accuracy isn’t the real problem. Instead, nonsmooth gradients are. A learned model can look “perfect” in prediction, yet completely break when used for planning or control. We introduce Smooth Neural Surrogates (SNS), a method for learning robot dynamics that remain optimization-friendly even during difficult-to-model events like contact. Combined with heavy-tailed (Cauchy) learning, this lets model predictive control work reliably where standard neural models break. In zero-shot locomotion tasks, SNS yields ~10–50% lower cost in simple tasks, and in hard regimes flips 0% successes to 100%, with ~2–50× lower cost. If you care about robotics, control, model-based RL, or differentiable optimization, this work argues that smoothness is a key missing factor for the learned model to be used for planning and control. Awesome work led by our PhD student Sam Moore from our Duke General Robotics Lab (https://lnkd.in/e2JjXfHz). - Video: https://lnkd.in/eiVrBcD8 - Website: https://lnkd.in/esrZakqk

How can multiple robots carry large objects together without communication or centralized control? 🔗 https://lnkd.in/gTXJD-7n This paper introduces a decentralized approach called Pinch-Lift-Move (PLM) for cooperative transport using quadruped robots with arms. Instead of relying on rigid mechanical coupling or explicit coordination, each robot learns policies that allow them to coordinate through physical interaction and local observations. ■ Key ideas: • A hierarchical policy separating locomotion and arm control • A novel “constellation reward” encouraging robots to behave as if rigidly attached to the object • Decentralized execution with shared policies Results show that policies trained with just two robots scale up to teams of ten, and generalize to different object shapes such as logs, barrels, and couches. This is an interesting step toward scalable multi-robot collaboration in the real world. ==== 『複数の四足歩行ロボットが協力して大きな物体を運ぶ研究』 🎥 https://lnkd.in/gGrd4FDz 通信や中央制御なしでも、ロボットは接触力と局所情報だけで協調し、Pinch-Lift-Moveの動作で物体を搬送する。 #quadrupedal #robot #ReinforcementLearning #EmbodiedAI #transport #MultiRobotSystems #DRAIL

Discrete-Time Hybrid Automata Learning: Legged Locomotion Meets Skateboarding https://lnkd.in/e8ifhKuH DHAL (Discrete-Time Hybrid Automata Learning) is a novel reinforcement learning framework that enables quadruped robots to perform hybrid dynamic tasks like skateboarding. Unlike traditional model-based methods that rely on predefined gaits or model-free RL that struggles with mode-switching, DHAL effectively handles discrete contact events and continuous motion. This allows the robot to adapt its locomotion smoothly in real-world environments. - Explicit Mode-Switching: Learns discrete mode transitions without needing pre-segmented trajectories. - Hybrid Dynamics Integration: Models contact-guided motions, improving stability on non-traditional terrains. - Sim-to-Real Transfer: Achieves real-world skateboarding with robust motion adaptation. Features - Hybrid Automata for Mode Identification Replaces manual gait segmentation with a discrete-time mode selector. Dynamically switches between gliding, pushing, and airborne phases. - Multi-Critic Reinforcement Learning Uses separate critics for gliding, pushing, and sim-to-real adaptation. Ensures smooth transitions and optimal contact management. - Beta Distribution Policy for Stable Control Improves action-space efficiency over traditional Gaussian policies. Reduces unstable movements and enhances real-world applicability. Implementation - Simulated Training with Hybrid Dynamics Train reinforcement learning (RL) policies to model contact events and mode transitions. Incorporate multi-critic training for stable motion learning. - Mode Recognition & Control Optimization Use a hybrid automata network to classify robot states into different motion phases. Train policies to optimize trajectory consistency and energy efficiency. - Real-World Deployment & Fine-Tuning Deploy policies on a quadrupedal robot (Unitree Go1). Adjust control signals in response to environmental disturbances. Challenges - Handling Hybrid Transitions – Overcomes RL's difficulty with contact phase switching using hybrid automata modeling for smooth mode transitions. - Sim-to-Real Transfer – Enhances real-world performance with multi-critic learning and beta distribution policies for robustness. - Efficient Exploration – Tackles sparse rewards by optimizing multi-task rewards to balance gliding and pushing. Performance in Apps - Achieved real-world skateboarding with Unitree Go1. - Legged Locomotion: Enhances hybrid terrain traversal for quadrupeds. - Assistive & Service Robotics: Enables adaptive robot motion in complex environments. DHAL provides a groundbreaking approach to hybrid locomotion, allowing legged robots to master non-standard mobility tasks, paving the way for more versatile robotic applications in urban, industrial, and outdoor environments. Follow me to know more about ML, AI and Robotics

It is difficult to get robots to be both precise and general. We just released a new technique for precise manipulation that achieves millimeter-level precision while being robust to large visual variations. The key is a careful combination of visuo-tactile learning and RL. The insight here is: vision and tactile are complementary. Vision is good at spatial, semantic cues, while touch excels at local contact feedback. ViTaL is a recipe to combine the two to enable precise control at >90% success rates even in unseen environments. For the full paper, videos and open-sourced code: https://lnkd.in/eAfhz8sE This work was led by Zifan Zhao & Raunaq Bhirangi, and a collaboration with Siddhant Haldar & Jinda Cui.

🔧 Precise manipulation meets generalization -- with just 32 demos and 45 minutes of interaction. Robots are getting better at learning from large-scale data -- just like we’ve seen in vision and language. But when it comes to precise tasks like inserting plugs, swiping cards, putting keys in locks or plugging USBs, scale alone isn’t enough. These contact-rich tasks demand millimeter-level accuracy, and collecting diverse, high-quality data is difficult. This leads to an unwanted tradeoff: generalization vs precision. We introduce VisuoTactile Local (ViTaL) policies -- a framework that leverages the complementary strengths of vision and touch to achieve generalizable, precise control for contact-rich manipulation. Our framework has two components: 🧠 Global policy (e.g., a pretrained VLM) handles coarse semantic localization. ✋ Local ViTaL policy takes over for the last-mile of precise, contact-rich execution. 💥 With just 32 demos per task and 45 min of real-world RL, ViTaL achieves >90% success on 4 contact-rich tasks -- inserting plugs, swiping cards, putting keys in locks and plugging USBs -- in cluttered, unseen environments. ViTaL policies can be trained in the lab and deployed in kitchens, homes and offices without any retraining! So how do you train a ViTaL policy? Two simple steps: 1️⃣ Behavior Cloning with semantic augmentations for robust visual generalization. This policy excels at reaching, but fails about ~50% of the time at the contact-rich portion of the task. 2️⃣ Visuotactile Residual RL effectively leverages tactile feedback for offset-based refinement, while maintaining the generalizability of the behavior cloning phase. 🔑 Key insights: 1️⃣ Tactile sensing is critical -- removing it drops performance by ~40%. 2️⃣ Egocentric vision offers consistent spatial context tied to the robot’s frame, enabling deployment on new robots. 3️⃣ Semantic augmentations improve generalization under scene and spatial variations. 4️⃣ Residual RL with strong visual encoders can boost task performance while preserving robustness. This work would not be possible without Zifan Zhao's relentless pursuit of precise policies that actually generalize, Siddhant Haldar's invaluable insights on policy learning and residual RL, and consistent feedback from Jinda Cui and Lerrel Pinto. For more details and videos: https://lnkd.in/eXP5xTht Check out our paper for a comprehensive ablation study: https://lnkd.in/e-wARCsB Open-source code: https://lnkd.in/ezTYsD4Q

Dexterity is coming closer! 🪬 What a start to the new year. During CES, Sharpa announced a tactile VLA for “last millimeter” manipulation called CraftNet. Sharpa says the real bottleneck in robotics is hands. Robots can dance, box, and run. But the moment they touch an object, they become clumsy, because most policies are trained on trajectories with no force or tactile feedback. Their answer is CraftNet, a hierarchical VTLA model (Vision–Tactile–Language–Action) built specifically for fine manipulation. Their architecture splits control into 3 layers: → System 2 (Reasoning Brain): vision + language planning (~1 Hz) → System 1 (Motion Brain): approach + pre-contact motion (~10 Hz) → System 0 (Interaction Brain): tactile + force-based micro-control during contact (~100 Hz) That last part is the key. They’re explicitly targeting what they call the “Last Millimeter Challenge”, those tiny contact adjustments humans do automatically: tightening grip, sliding fingers, re-grasping, correcting force. CraftNet tries to turn widely available video, sim or tele-op data into manipulation data with tactile signals, so training doesn’t get stuck in the “data drought.” One of the most awesome robotics dexterity showcases 🤯 ~~ ♻️ Join the weekly robotics newsletter, and never miss any news → ziegler.substack.com