Simulation and Modeling of Robots

Build your first robot in simulation! 👾 📌 If you’re self-learning robotics, this is genuinely one of the better repos to save for later. NVIDIA Robotics released a "Getting Started with Isaac Sim" tutorial series covering everything from building your first robot to hardware-in-the-loop deployment. What's inside? → Building Your First Robot Explore the Isaac Sim interface, construct a simple robot model (chassis, wheels, joints), configure physics properties, implement control mechanisms using OmniGraph and ROS 2, integrate sensors (RGB cameras, 2D lidar), and stream sensor data to ROS 2 for real-time visualization in RViz. → Ingesting Robot Assets Import URDF files, prepare simulation environments, add sensors to existing robot models, and access pre-built robots to accelerate development. → Synthetic Data Generation Learn perception models for dynamic robotic tasks, understand synthetic data generation, apply domain randomization with Replicator, generate synthetic datasets, and fine-tune AI perception models with validation. → Software-in-the-Loop (SIL) Build intelligent robots, implement SIL workflows, use OmniGraph for robot control, master Isaac Sim Python scripting, deploy image segmentation with ROS 2 and Isaac ROS, and test with and without simulation. → Hardware-in-the-Loop (HIL) Understand HIL fundamentals, learn NVIDIA Jetson platform, set up the Jetson environment, and deploy Isaac ROS on Jetson hardware. The progression makes sense: start with basics (build a robot), add perception (sensors and data), generate training data (synthetic generation), develop software (SIL), then deploy to hardware (HIL). Each module builds on the previous one. For robotics teams, this is the path to faster iteration. Simulate first, validate in software-in-the-loop, generate synthetic training data at scale, then deploy to hardware with confidence. 🎓 If this helps at least one engineer to become more fluent in the world of robotics, means a lot to me! 🫶🏼 Here's the course (it's free): https://lnkd.in/dRYdkmdi ~~ ♻️ Join the weekly robotics newsletter, and never miss any news → ziegler.substack.com

2–6 DOF Robotic Manipulators Trajectory Tracking using PID in MATLAB ➡ Simulation of 2-DOF to 6-DOF robotic manipulators ➡ Detailed modeling of serial manipulators including UR5 ➡ Forward & Inverse Kinematics implementation for all DOF systems ➡ PID-based joint control for smooth and stable motion ➡ Trajectory tracking: Circle, Rectangle, and Infinity (∞) paths ➡ Real-time 3D visualization and animation in MATLAB ➡ Modular and well-structured code for scalability and learning ✨ Why this matters: Trajectory tracking is a fundamental problem in robotics, where a manipulator must precisely follow a desired path while maintaining stability and accuracy. This becomes increasingly complex as the number of degrees of freedom increases due to nonlinear kinematics, joint coupling, and control challenges. This project demonstrates how classical control techniques like PID can be effectively applied to multi-DOF robotic systems to achieve smooth and reliable motion. By integrating kinematic modeling with control strategies, the system reflects real-world industrial applications where robotic arms are required to perform precise tasks such as assembly, welding, and pick-and-place operations. 📊 Key Highlights: ✔ Complete kinematic modeling (FK & IK) for 2–6 DOF manipulators ✔ PID-based trajectory tracking for accurate motion control ✔ Implementation of multiple trajectories (circle, rectangle, infinity) ✔ Real-time simulation and visualization in MATLAB ✔ Clean and reusable code structure for educational use ✔ Industrial-level modeling with UR5 6-DOF manipulator 💡 Future Potential: This framework can be extended to: ➡ Advanced control (Adaptive, MPC, Fuzzy, AI-based control) ➡ Obstacle avoidance and path planning ➡ Integration with ROS 2 for real robot deployment ➡ Dynamic modeling and torque control ➡ Digital twin and industrial automation systems 🔗 For students, engineers & robotics enthusiasts: This project provides a complete hands-on approach to understanding robotic manipulators, control systems, and trajectory planning. It is ideal for learning how robotic arms achieve precise motion in real-world applications. 🔁 Repost to support robotics innovation & engineering learning! #Robotics #MATLAB #PIDControl #RobotManipulators #UR5 #ControlSystems #Automation #Mechatronics #EngineeringProjects #Simulation #STEM #EngineeringEducation

Big shift in robotics: NVIDIA just open-sourced Isaac Sim and Isaac Lab. Isaac Sim has already been a cornerstone for high-fidelity robotics simulation—RTX-accelerated physics, realistic lidar/camera simulation, domain randomization, ROS/URDF support, and synthetic data pipelines. Now, it’s all on GitHub with full source access. But the real multiplier? The release of Isaac Lab—a modular, open reinforcement learning and robot control framework built directly on top of Isaac Sim. It comes with ready-to-use robots (Franka, UR5, ANYmal), training loops, and environments for manipulation, locomotion, and more. What’s different now: *You’re no longer limited to APIs—developers can modify physics, sensors, and control logic at the source level. *Isaac Lab provides a training-ready foundation for sim-to-real robotics, speeding up learning pipelines dramatically. *Debugging, benchmarking, and custom integrations are now transparent, flexible, and community-driven. *Collaboration across research and industry just got easier—with reproducible environments, tasks, and results. We’ve used Isaac Sim extensively, and this open-source release is going to accelerate innovation across the robotics community. GitHub: https://lnkd.in/gcyP9F4H

Exciting updates on Project GR00T! We discover a systematic way to scale up robot data, tackling the most painful pain point in robotics. The idea is simple: human collects demonstration on a real robot, and we multiply that data 1000x or more in simulation. Let’s break it down: 1. We use Apple Vision Pro (yes!!) to give the human operator first person control of the humanoid. Vision Pro parses human hand pose and retargets the motion to the robot hand, all in real time. From the human’s point of view, they are immersed in another body like the Avatar. Teleoperation is slow and time-consuming, but we can afford to collect a small amount of data.  2. We use RoboCasa, a generative simulation framework, to multiply the demonstration data by varying the visual appearance and layout of the environment. In Jensen’s keynote video below, the humanoid is now placing the cup in hundreds of kitchens with a huge diversity of textures, furniture, and object placement. We only have 1 physical kitchen at the GEAR Lab in NVIDIA HQ, but we can conjure up infinite ones in simulation. 3. Finally, we apply MimicGen, a technique to multiply the above data even more by varying the *motion* of the robot. MimicGen generates vast number of new action trajectories based on the original human data, and filters out failed ones (e.g. those that drop the cup) to form a much larger dataset. To sum up, given 1 human trajectory with Vision Pro  -> RoboCasa produces N (varying visuals)  -> MimicGen further augments to NxM (varying motions). This is the way to trade compute for expensive human data by GPU-accelerated simulation. A while ago, I mentioned that teleoperation is fundamentally not scalable, because we are always limited by 24 hrs/robot/day in the world of atoms. Our new GR00T synthetic data pipeline breaks this barrier in the world of bits. Scaling has been so much fun for LLMs, and it's finally our turn to have fun in robotics! We are creating tools to enable everyone in the ecosystem to scale up with us: - RoboCasa: our generative simulation framework (Yuke Zhu). It's fully open-source! Here you go: http://robocasa.ai - MimicGen: our generative action framework (Ajay Mandlekar). The code is open-source for robot arms, but we will have another version for humanoid and 5-finger hands: https://lnkd.in/gsRArQXy - We are building a state-of-the-art Apple Vision Pro -> humanoid robot "Avatar" stack. Xiaolong Wang group’s open-source libraries laid the foundation: https://lnkd.in/gUYye7yt - Watch Jensen's keynote yesterday. He cannot hide his excitement about Project GR00T and robot foundation models! https://lnkd.in/g3hZteCG Finally, GEAR lab is hiring! We want the best roboticists in the world to join us on this moon-landing mission to solve physical AGI: https://lnkd.in/gTancpNK

A few years ago, I learned the hard way that jumping straight into hardware, sensors, motors, and wiring can lead to costly mistakes and late-night headaches. That’s when I discovered the true importance of #simulation in robotics and engineering. During the early phase of my final-year thesis, I spent weeks recreating our school cafeteria with Iman Tokosi in Blender, exporting it as an SDF model and loading it into Gazebo using #ROS2. Suddenly, I could drive a virtual robot through aisles and around tables without the fear of damaging anything real. It was challenging and eye-opening, and it saved me countless hours and resources. Then came the moment that changed everything: integrating #SLAM so the robot could build its own map while moving, and setting up #Nav2 to let it plan and follow paths autonomously. Watching it navigate the environment with precision and independence was a powerful confirmation that the system worked. Now, imagine a world where every structure, product, and system is simulated down to the smallest detail. The result? Reduced costs, faster development, increased reliability, enhanced safety, and stronger adherence to standards. Some may still view simulation as “just for show,” but I’ve experienced firsthand that it’s the foundation of true innovation. Are you leveraging simulation in your next robotics or engineering project? Let’s connect and exchange ideas!

What if a robot could simulate the physical world from a single image. [📍Bookmark Paper & GitHub for later] PointWorld-1B from Stanford and NVIDIA is a large 3D world model that predicts how an entire scene will move, given RGB-D input and robot actions. The key idea is simple but powerful: Actions are not joint angles❗️ They are 3D point flows sampled from the robot’s own geometry. The model reasons in the same space where physics actually happens. • State and action are unified as 3D point trajectories. • One forward pass predicts full-scene motion for one second. • No object masks, no trackers, no material priors. • Trained on ~500 hours of real and simulated robot interaction data. • Micrometer-level trajectory error, thinner than a human hair. • Works across embodiments, from single arm to bimanual humanoid. The model is then used inside an MPC planner to push objects, manipulate cloth, and use tools, all zero-shot, from a single fixed camera and without finetuning. This feels like a shift from “learning policies” to “learning physics in 3D”. Thanks for sharing, @wenlong_huang 📍Project point-world.github.io 📍Paper arxiv.org/abs/2601.03782 📍GitHub https://lnkd.in/dqsjUTxg (will be published soon) —— Weekly robotics and AI insights. Subscribe free: scalingdeep.tech

𝗛𝗨𝗠𝗔𝗡𝗢𝗜𝗗 𝗥𝗢𝗕𝗢𝗧𝗜𝗖𝗦: 𝗪𝗵𝗲𝗿𝗲 𝗦𝗶𝗺𝘂𝗹𝗮𝘁𝗶𝗼𝗻 𝗕𝗲𝗴𝗶𝗻𝘀 𝗮𝗻𝗱 𝗟𝗲𝗮𝗿𝗻𝗶𝗻𝗴 𝗔𝘄𝗮𝗸𝗲𝗻𝘀 Deep Learning in 3D Simulation is not a lab exercise. It is the moment we begin to teach machines how to exist. Not to repeat motions. Not to merely follow code. But to learn, adapt, balance, reason, and act with purpose. In my project we are not just building robots. We are building a new class of intelligence that experiences the world before it ever touches reality. In these simulation environments, gravity does not remain constant. Terrain does not always cooperate. Obstacles change shape. Sensors lie. Friction shifts. And the humanoid must still stand, walk, grasp, adjust, optimize, and choose its next step. Domain randomization, reinforcement learning, hierarchical policies, and graph neural dependencies no longer sound like academic theory. They become survival tools. Machines begin to develop strategies. They learn how to carry payloads across unstable rubble. They learn energy discipline when battery is low and temperature is high. They learn trajectory planning not as geometry, but as survival logic. When you combine photorealistic environments from Isaac Sim, contact-perfect physics in MuJoCo, embodied navigation in Habitat, and emergent behavior in Unity, you begin to see something different. You see machines build experience. You see memory. You see policy retention. You see adaptation. You see the beginning of abstract perception where simulation is not just testing, but education. The difference between teaching a robot how to walk, and letting it discover how to navigate a collapsing environment with intelligence and intent. This is where humanoid robotics becomes human oriented. Robots that can open doors without templates. Carry supplies without pre-programmed routes. Coordinate convoys. Assist in evacuation. Make real time physical decisions aligned with mission objectives, not static instructions. Simulation gives us time compression. We can give a single humanoid what would have taken humans years of trial. We can compress thousands of failures into one informed policy. This is how we transform capability. Not automation. Cognitive autonomy. Not motion planning. Motion intelligence. Not digital twins. Learning twins. We are building humanoids that do not just survive the environment. They learn from it. If you are in advanced simulation, deep learning pipelines, physics engines, reinforcement learning, biomechanics, embodied cognition, ROS2, Isaac Sim, MuJoCo, Omniverse, Habitat, Unity, Unreal, LLM integration, perception or policy optimization… Then we should not be working apart. We should be building this together. And for those ready to build the next generation of thinking humanoids Singularity Systems is now accepting collaborators, researchers, engineers, architects, and visionaries. Let’s teach machines how to exist. #changetheworld #3D #unity

The NVIDIA robotics and simulation teams have worked for several years on simulation and sim-to-real for contact-rich manipulation, most notably robotic assembly. We developed simulation technology in Factory (RSS 2022), demonstrated sim-to-real in IndustReal and AutoMate (RSS 2023 and 2024), explored multimodal policy learning in FORGE and TacSL (RA-L and T-RO 2025), and explored skill retrieval in SRSA (ICLR 2025) and assembly generation in MatchMaker (ICRA 2025). If you'd like to learn a bit more, check out the NVIDIA R2D2 digest here! https://lnkd.in/eA9fVhX8 Despite the progress, we're just getting started. Some questions on our mind: - How do we push policy success rates and cycle times well beyond research standards, and much closer to what industry needs? - How do we learn generalist policies for assembly, and/or train VLAs on this domain? - How do we train adaptive, robust behaviors for truly complex assembly problems, like those found in the electronics industry?

Most RL tutorials stop at simulation or show impressive hardware results without explaining the engineering process that made them work. This guide bridges the gap to real hardware with a complete working system – training code, hardware deployment, 3D models, trained checkpoints – and comprehensive documentation of the engineering methodology that made it work. You get the reward design process, sensor characterization approach, debugging frameworks, and decision-making that got RL working on a real robot. What could take you months of trial and error is compressed into a proven methodology you can follow in days. What You’ll Be Able to Do - Build accurate MuJoCo models that enable hardware transfer - Train RL policies that work on real robots, not just simulation - Systematically debug sim-to-real failures - Apply this methodology to more complex robots (humanoids, quadrupeds) https://lnkd.in/gWRmDxDs

Claude can now generate animations : Very helpful to understand topics like diffusion, flow matching etc. Made a free, open-source site(link in first comment) with interactive animations that explain how modern robot AI actually works. Pick a topic, click through the steps, see what's happening under the hood: → How VLA models turn camera frames into robot actions → Why sim-trained robots fail in the real world → How diffusion policy turns noise into smooth trajectories → Flow matching — the faster alternative powering π0 → Reward shaping tradeoffs that make or break RL → How video prediction drives robot planning There's also an interactive timeline of every major robot foundation model from CLIPort (2021) to π0.6 (2026). Everything runs in your browser. No installs, no sign-ups, no dependencies. Just HTML/CSS/JS. If you think a concept is missing, open an issue — or better, submit a PR.