Building Dynamic Camera Perspectives in Robotics

Imitation learning has seen great success, but IL policies still struggle with OOD observations such as changes in camera pose. We designed a 3D backbone, Adapt3R, that can combine with your favorite Imitation Learning algorithm to enable zero-shot generalization to unseen embodiments and camera viewpoints! Learning 3D representations is hard without 3D data 💡 The key idea is to use a 2D foundation model to extract semantic features, and use 3D information to localize those features in a canonical 3D space without extracting any semantic information from the 3D data. Adapt3r unprojects 2D features into a point cloud, transforms them into the end effector’s coordinate frame, and uses attention pooling to condense them into a single conditioning vector for IL. Notice that Adapt3R attends to the same points before and after the camera change! 2D features lifted into 3D are an effective representation for this scenario and Adapt3R makes good use of them! So, what did we observe empirically? - Adapt3R is just as proficient as RGB based baselines in case of in distribution evaluations. - But, Adapt3R is very good at embodiment transfers - Most importantly, Adapt3R handles viewpoint changes at test time! No More fixing the camera to match training distribution! Overall this means Adapt3R provides 3D representations as a drop in replacement for 2D RGB baselines. this was led by Albert Wilcox with help from Mohamed Ghanem, Masoud Moghani, Pierre Barroso, Benjamin Joffe and Animesh Garg Check out more at and play with the code in your next robot learning project. 🌐 Website: https://lnkd.in/dWneBJ5d 📄 Paper: https://lnkd.in/dvcbA_22 🖥️ Code: https://lnkd.in/dFKYjym7

In the world of #binpicking and Vision-Guided Robotics (VGR), a single perspective isn't always enough. Traditional single-scan setups often struggle with "blind spots," but the Photoneo, now part of Zebra Technologies' MultiView technology, is changing the game, engineered to bypass the physical limitations of a single-scan perspective. 🔴 Why MultiView? 🔵 By combining 3D data from multiple viewpoints into a single, high-resolution 3D point cloud, we eliminate the hurdles that stop most automation lines: ✅ Occlusions: See what others miss by looking around obstacles. ✅ Complex Surfaces: Perfect for thin, reflective, or irregularly shaped parts. ✅ Large Objects: Achieve full coverage of oversized components without sacrificing detail. 💡 Two Ways to Deploy: 👉 Static: Utilizing multiple PhoXi 3D Scanners for rock-solid reliability in fixed environments: https://lnkd.in/eaNPAdr6 👉In Motion: Using the MotionCam-3D in a hand-eye configuration to capture data dynamically while the robot moves: https://lnkd.in/eT5B7Bbz The result? An unmatched localization success rate, even in the most complex industrial scenarios. If you want to increase your throughput and decrease error rates, it's time to look at the bigger picture. Stop settling for one point of view. See everything. Universal Robots #digitaltwin #physicalAI #3DVision #Robotics #Automation #BinPicking #Photoneo #ZebraTechnologies #SmartManufacturing #VGR #Innovation #automotive #Tier1 #3Dscanning #3Dcamera #machinetending

Vision language action models finally get to look around on their own. Through the paper SaPaVe, researchers allowed robots to actively move their head-mounted camera to find objects completely out of view. Instead of the usual fixed near-optimal camera setup, SaPaVe adds a 2-DoF active head (pitch + yaw) that the model learns to control semantically from language instructions alone. They accomplished this using: - A two separate action spaces, splitting the final model output into two MLP heads, one for camera movement (2-DoF) and one for manipulation (26-DoF), so the two skills don't interfere during training - ActiveViewPose-200K, a synthetically generated dataset of 200k image-instruction-camera movement triplets built cheaply in simulation A two-stage training strategy where the first train only the camera head on ActiveViewPos and the second trains the manipulation head on real robot data while mixing in camera examples to prevent forgetting - Spatial Knowledge Injection via MapAnything, feeding 3D geometry (depth, camera poses) into the action decoder so the policy stays stable even with viewpoint changes (Day 2 of highlighting interesting CVPR 2026 papers about VLAs)

[Multi-Camera View Scaling for Data-Efficient Robot Imitation Learning]: Exploiting scene diversity through camera viewpoints to enhance imitation learning Arxiv: https://lnkd.in/eivNYajX Project: [Link not provided] 🔁 At a Glance 💡 Goal: Improve imitation learning efficiency and generalization by exploiting inherent scene diversity through camera view scaling. ⚙️ Approach: - Pseudo-Demonstrations: Generate multiple viewpoints from expert trajectories using synchronized cameras. - Action Space Analysis: Study how different robot action representations interact with view scaling. - Multiview Action Aggregation: Combine multiple camera inputs during inference to enhance policy robustness. 📈 Impact (Key Results) 🧪 Data Efficiency: - Significant improvement in success rates across simulation and real tasks. - Pseudo-demonstrations from multiple views outperform single-view baselines. 🔄 Generalization: - Policies trained with multiple views generalize better to unseen camera angles. - Multiview inference boosts performance without architectural changes. 🤖 Practicality: - Minimal hardware add-on; scales existing demonstration datasets. - Compatible with various imitation learning algorithms. 🔬 Experiments 🧪 Benchmarks: robomimic, real-world water pouring task. 🎯 Tasks: Manipulation and pouring. 🦾 Setup: Simulation and physical robot with two cameras. 📐 Inputs: RGB images from multiple views → robot actions. 🛠 How to Implement 1️⃣ Install multiple synchronized cameras during demonstration collection. 2️⃣ Convert expert trajectories into pseudo-demonstrations for each view. 3️⃣ Train a visuomotor policy with the expanded dataset. 4️⃣ During inference, optionally perform multiview aggregation. 5️⃣ Deploy the policy with enhanced viewpoint robustness. 📦 Deployment Benefits ✅ Increased data efficiency with existing demonstrations. ✅ Improved viewpoint invariance. ✅ Seamless integration with pre-existing algorithms. ✅ Cost-effective hardware setup. 📣 Takeaway Leveraging inherent scene diversity via camera view scaling significantly boosts robot imitation learning. This approach maximizes data utility without costly scene re-collection. Multiview supervision is a scalable solution for robust, generalizable policies. Focusing on viewpoint diversity unlocks untapped potential in visual imitation learning. Follow me to know more about AI, ML and Robotics!