Autonomous Vehicle Engineering

One of the key challenges of autonomous driving is scalably handling the complexity of driving scenarios, where traffic rules, city environments, and vehicles/pedestrians can interact in a myriad of possible ways. It’s not tractable to create hand-crafted rules that handle every case, so instead we rely on the power of “next frame prediction” in a compact world representation. Here the world representation is semantic segmentation, which captures the essence of what’s happening around a vehicle, and can be stably computed in real-time using Helm.ai’s production grade perception stack. One example of a set of complex scenarios is an intersection with traffic lights, which presents a large number of possibilities that an autonomous vehicle must navigate safely. To tackle this challenge, we added traffic light segmentation and traffic light state to our world model representation, and trained a foundation model to predict what might happen next based on an input sequence of observed segmentations. Our foundation model learned in a fully unsupervised way from real driving data the relationship between traffic light state and what the vehicles/agents on the road should do in various contexts. The result is an ability to forecast a wide variety of scenarios of interaction between traffic lights, intersection geometry, vehicles, and pedestrians that are consistent with potential real world scenarios, including predicting the paths of the ego vehicle and the other agents. In our latest demo, our intent and path prediction models predict 9 seconds into the future using 3 seconds of observed driving data, at 5 frames per second. This prediction capability includes learned human-like driving behaviors, such as intersection navigation, interaction with green and red lights, yielding to oncoming traffic before turning, and keeping a safe distance to other vehicles. Our foundation models are able to predict these future behaviors and plan safe paths by scalable learning from real driving data, without any hand crafted rules nor traditional simulators. Stay tuned for upcoming updates as we continue to expand our unified approach to ADAS through L4 autonomous driving by enriching the world model representation and scaling up our predictive DNNs. #helmai #generativeai #selfdrivingcars #artificialintelligence #ai #autonomousdriving #adas #computervision 

On the heels of the Alpamayo announcement — NVIDIA’s fully open ecosystem for accelerating the development of reasoning-based autonomous vehicles — I’m excited to share our latest advances in researching reasoning-based Physical AI models. Starting with Latent‑CoT‑Drive (LCDrive), a novel approach that learns to reason in a *latent* action-aligned space for end-to-end driving decision-making. Traditional vision-language-action models rely on natural language for chain-of-thought reasoning — but is language the best medium for encoding driving decisions? In our paper, we explore this question and introduce a latent representation that integrates both action proposals and predictions of future outcomes, enabling richer reasoning and improved performance. 🔍 Key Contributions - Latent reasoning for driving: LCDrive rethinks reasoning in vision–language–action (VLA) models using latent chain-of-thought tokens aligned with driving actions and a latent world model. - Effective training framework: Combines latent CoT cold-start, world model training, and closed-loop reinforcement learning, tailored for latent reasoning models. - Empirical gains: Shows faster inference and higher driving quality compared to non-reasoning and text-reasoning baselines. This work shows that latent reasoning provides a compelling representation for reasoning-based VLA models. 📄 Full paper here: https://lnkd.in/ejsZnwkw #AutonomousVehicles #AutonomousDriving #PhysicalAI #ReasoningAI #Alpamayo NVIDIA AI NVIDIA DRIVE

Imagine a vehicle that can "see" the world around it with precision and detail surpassing human capability. Autonomous vehicles achieve this feat thanks to an orchestra of sophisticated sensors, including cameras, LiDAR, radar, and ultrasonic sensors. These sensors capture a torrent of data, from images and videos to light pulses and radio waves, painting a detailed picture of the environment. But data alone isn't enough. This is where the magic of deep learning and computer vision comes in. The first step towards autonomous navigation is environment perception. Autonomous vehicles rely on a multitude of sensors to capture information about the world around them, including: 1) Cameras: Capture images and videos of the surroundings, essential for detecting objects like vehicles, pedestrians, traffic signs, and lane markings. 2) LiDAR (Light Detection and Ranging): Emits laser light pulses and measures the return time to create precise 3D maps of the environment, including the distance and depth of objects. 3) Radar: Detects moving objects using radio waves, proving useful in low-visibility conditions like rain or fog. 4) Ultrasonic Sensors: Measure the distance to nearby objects, primarily used for low-speed collision avoidance. 5) Planning and Decision-Making: The Brain of Autonomous Vehicles. Once the environment has been perceived, autonomous vehicles need to make intelligent decisions to navigate safely and efficiently. This is where route planning and decision-making come into play. Machine Learning algorithms play a critical role in these tasks: A) Route Planning: Determine the optimal route to reach the destination, considering factors like traffic, traffic regulations, and road conditions. B) Predicting the Behavior of Other Users: Anticipate the actions of other vehicles, pedestrians, and cyclists to avoid collisions and dangerous maneuvers. C) Real-Time Decision Making: Adapt the driving plan to unexpected events, such as sudden braking, lane changes, or pedestrians crossing the street. D) Continuous Learning: Improving with Experience E) Adapt to New Situations: Learn from experience and adjust their behavior based on different driving conditions, climates, and environments. F) Update Maps and Models: Incorporate new information about the surroundings, such as changes in road infrastructure or new traffic signs. G) Personalize the Driving Experience: Tailor the driving style to user preferences, prioritizing safety, efficiency, or comfort. Autonomous vehicles, powered by Machine Learning, have the potential to revolutionize the way we move. They offer the promise of safer, more efficient, and sustainable mobility, reducing traffic accidents, congestion, and emissions. However, the technological and regulatory challenges are significant. Continued research and development, along with an ethical and responsible approach, are essential to ensure autonomous vehicles become a reality that benefits all of society.

Autonomous vehicle technology represents a paradigm shift in societal mobility, offering the potential to eliminate human error and provide critical independence to the elderly and mobility-impaired populations. Despite significant progress, the "long-tail" of rare, high-risk edge cases remains a primary barrier to safe, full-scale deployment. In a new paper (https://lnkd.in/eKZp6tpa) with my colleagues (Dr. Edward Griffor, Munawar Hasan, Mahima Arora, Honglan Jin, Pavel Piliptchak, Thoshitha Gamage) we approach this problem by evaluating perception performance using predictive sensitivity quantification based on an ensemble of models, capturing model disagreement and inference variability across multiple models, under adverse driving scenarios in both simulated environments and real-world conditions. We propose a notional architecture for assessing perception performance that comprehends multiple input sources and extensible AI architecture that provides detection and classification as outputs along with predictive sensitivity and post-processing. Diminished lighting conditions, e.g., resulting from the presence of fog and low sun altitude, are seen to have the greatest impact on the performance of the perception models. Additionally, adversarial road conditions such as occlusions of roadway objects increase perception sensitivity and model performance drops when faced with a combination of adversarial road conditions and inclement weather conditions. Also, it is demonstrated that the greater the distance to a roadway object, the greater the impact on perception performance, hence diminished perception robustness. This work is only the first step in a bigger effort at the National Cybersecurity Center of Excellence (https://lnkd.in/eYjDs5tM) where we continue the work by developing a public dataset spanning multiple modalities (camera, LiDAR, radar) and a testbed with difficult-to-handle and adversarial road/traffic conditions with the goal of improving autonomous vehicles and accelerating their safe deployment. We are open to collaboration, join us at NCCoE.

🚘 Roadmap to Mastering Autonomous Vehicle Systems (AVS) The field of Autonomous Vehicles is one of the fastest-growing areas in robotics and AI — and having a clear learning roadmap is essential for anyone who wants to enter the industry. 🧠 1. Foundations of AV Engineering Robotics mathematics (linear algebra, calculus, geometry) Probability & statistics Control systems (PID, state-space, MPC) Vehicle dynamics & kinematics 🔍 2. Perception Computer Vision: OpenCV, deep learning for detection & segmentation Sensors: LIDAR, RADAR, stereo/depth cameras, IMU, GNSS Sensor Fusion: EKF/UKF, multi-sensor fusion pipelines 🗺️ 3. Localization & Mapping SLAM (LIDAR-based, Visual SLAM, RGB-D SLAM) Odometry (wheel, visual, inertial) HD mapping & map representation 🧭 4. Planning & Decision Making Path planning: A*, D*, RRT, lattice planning Behaviour planning & state machines Trajectory generation & optimization ⚙️ 5. Control Longitudinal control (speed/throttle) Lateral control (pure pursuit, Stanley, MPC) Real-time embedded control 🛠️ 6. Systems & Middleware ROS / ROS2 CAN bus, automotive communication standards Real-time computing & simulation 🧪 7. Simulation & Testing CARLA, Gazebo, LGSVL Scenario testing Data logging & replay 🚀 8. Deployment Embedded Linux, NVIDIA Jetson Optimization for real-time performance Safety, redundancy, and fail-safe design Autonomous Vehicles combine AI, robotics, and embedded systems into one complex, exciting domain. If you commit to this roadmap step by step, you’ll be ready to build real AV systems — from perception all the way to full autonomy. ENG_Alaa Soliman #AutonomousVehicles #Robotics #AI #ComputerVision #ROS #SLAM #DeepLearning #EmbeddedSystems #Mechatronics #AVS #Simulation #PathPlanning #ControlSystems

I'm delighted to share a newly released technical report from our engineering team on the latest advances in world models. XPENG X-World is a physics AI system that can "think through" driving scenarios. It simulates and predicts how road conditions will evolve seconds into the future, based on real-time road environments and driving maneuvers. XPENG X-World has already become a foundational enabler across key pillars of our autonomous driving development, including closed-loop simulation testing, reinforcement learning, and targeted data generation. For example, leveraging its core capability of controllable generation, X-World is focused on improving the performance of our VLA 2.0 in challenging scenarios such as sudden pedestrian dart-outs at intersections and hesitation during lane changes in congested traffic. Meanwhile, X-World generates region-specific overseas driving data for model training, accelerating the global rollout of XPENG's autonomous driving technology. Please dive into the full details in our technical report: https://lnkd.in/ecV3N2jM

In this demo we extend our prior work on obstacles avoidance at aggressive speeds, showcasing our Thar based autonomous vehicle navigating at near drift speeds, progressing towards our endeavour of Level-5 autonomy. Our autonomous vehicle at Swaayatt Robots (स्वायत्त रोबोट्स) was tasked with avoidance of traffic cones on the road, placed in a zig-zag fashion, at aggressive speeds. The location of the marked cones was not known to the planner beforehand. The #autonomousdriving task, i.e., motion planning (time parametrized trajectory computation) and decision making, was made even more challenging by restricting the AI agents to not act on obstacles unless they are within 24m radius. Level-5 #autonomousvehicles should be able to react quickly to overtake, or to avoid, any sudden unforeseeable obstacle or pedestrian on the road to avoid fatalities -- a capability demonstrated by our novel motion planning and decision making algorithmic framework over here. Our previous demo showcased our Bolero based platform consistently keeping speeds beyond 45 KMPH for most part, slowing down to only 39 KMPH at one point. Given Thar has lesser body roll, our framework successfully kept speeds well above 47 KMPH (even at the points of avoidance of obstacles), with speeds reaching as high as 55 KMPH. A typical human driver would feel uncomfortable at speeds beyond 40 KMPH in such as scenario. The entire algorithmic framework with 5 classical (one #reinforcementlearning-) agents , runs at 800+ Hz on a regular i7 processor, single thread. This algorithmic framework is being further scaled up with end-to-end deep reinforcement learning, and will be showcased in the month of March. #deeplearning #machinelearning #motionplanning

Autonomous tech is absolutely on its way. There are teams and people like Philip Koopman and others deeply focused on AV safety. Regulators and industry folks are working on frameworks. But we do not yet have a mature safety or compliance framework for autonomous truck operations in the US. The rules are being figured out as the technology evolves. We see trucks with no drivers in some lanes, but these systems aren’t fail‑proof. In passenger vehicles and driver‑assisted systems, automation doesn’t eliminate crashes; in many cases, current automation systems are involved in incidents at higher rates than conventional human drivers on a per‑mile basis. Data from multiple sources shows autonomous or advanced driving systems at roughly 9+ crashes per million miles compared to 4 per million for human‑driven vehicles. Across thousands of reported incidents related to autonomous systems between 2019 and 2025, hundreds involved injuries and fatalities. We also see situations where automation fails to detect or respond correctly to real‑world hazards, things like emergency vehicles on the roadside, trailers turning into traffic, and stationary objects that a competent human driver would avoid. These are patterns that emerge in real‑world data. That’s not to say autonomous tech can’t be safer someday. But we are not there yet. Not even close. 👉We don’t have a robust, enforceable safety or compliance framework yet. Policymakers, DOT, and National Highway Traffic Safety Administration NHTSA, Federal Motor Carrier Safety Administration are working on this, but it’s still a patchwork and a work in progress. 👉Driverless test trucks are operating in limited corridors, not a national safe‑deployment model. The technology needs rigorous operational standards before full adoption. 👉Crash data from autonomous passenger and delivery vehicles shows there’s still a significant gap between aspiration and reality. We need transparency, accountability, and requirements that ensure safety before we hand over America’s highways to machines without a driver able to intervene. If you’re interested in understanding this technology start with the SAE International. That’s where the industry baseline lives, and SAE offers both free and paid resources to get up to speed. Innovation should improve safety, not outpace it. Autonomy is inevitable in one form or another but we’re still building the rulebook while the cars and trucks are already on the field. That’s a reality every carrier, regulator, and safety professional needs to understand. Until we have a solid compliance and safety structure in place not just pilot programs, driverless in name is not the same as safe in practice. #av #autonomousvehicles #riskmitigation #exposuremanagement

Fancy robot demos < 6 announcements from NVIDIA are driving the robotics future. While viral videos get the views, the real revolution is happening in the "brain" of the machine. NVIDIA Robotics & NVIDIA AI just released a suite of models focused on reasoning and validation that move us beyond simple motion. Here are the 6 key announcements: Physical AI & Humanoids ✦ Cosmos Reason 2: A new leaderboard-topping reasoning VLM that helps robots see, understand, and interact with high accuracy in the physical world. A warehouse robot that can look at a cluttered shelf and identify a specific damaged package hidden behind others, rather than just scanning barcodes. ✦Cosmos Transfer 2.5 & Predict 2.5: Leading models that generate large-scale synthetic videos across diverse environments to train AI before it touches reality. Training delivery drones to fly safely in heavy snow or smoke without risking a single physical crash during the learning process. ✦Isaac GR00T N1.6: An open reasoning VLA purpose-built for humanoids. It uses Cosmos Reason for better context understanding and full-body control. A humanoid robot carrying a tray of open liquids while stepping over cables on the floor, adjusting its balance in real-time to avoid spilling. Autonomous Vehicles (The Alpamayo Family) ✦Alpamayo 1: The first open, large-scale reasoning VLA for AVs. It enables vehicles to understand their surroundings and—crucially—explain their actions. An autonomous car that yields at a green light and explains it is waiting because it sees an ambulance approaching from a blind angle. ✦AlpaSim: An open-source simulation framework for closed-loop training and evaluation of these reasoning models across edge cases. Simulating millions of variations of a child running out from behind a parked truck to ensure the braking system reacts correctly every single time. Real-World Validation This isn't just theory. NVIDIA confirmed they have already tested this reasoning stack with Mercedes-Benz AG on European roads, and it will be coming to the U.S. soon. The new Mercedes CLA is set to be the first vehicle to use this next-gen, reasoning-based AV software. We are entering the era of reasoning-based automation. Comment "reason" down below and I will send you the links to these repositories and papers. 📩 PS: I break down the engineering behind projects like these every week. Join thousands of builders and researchers staying ahead of the Embodied AI curve. #NVIDIA #Robotics #PhysicalAI #CES2026 #FutureTech #Automation, #AutonomousVehicles #VLA #HuggingFace

NVIDIA's PhysicalAI Autonomous Vehicles Dataset The PhysicalAI-Autonomous-Vehicles dataset is one of the largest, most geographically diverse multi-sensor datasets ever released for AV research. And the numbers are staggering. Here's what we're talking about:  ↳ 1,727 hours of driving data across 25 countries and 2,500+ cities  ↳ 310,895 clips, each 20 seconds long  ↳ Seven camera views, 360-degree LiDAR, and up to 10 radar sensors per vehicle  ↳ Approximately 100TB of data What makes this particularly valuable is the geographic and environmental diversity. Half the data comes from across the United States, with the other half spanning 24 European countries. This captures wildly different traffic patterns, road types, weather conditions, and driving cultures in a single dataset. The data covers everything from clear highways to snow-covered residential streets, from empty rural roads to dense urban traffic. Weather conditions range from clear skies to fog, rain, and snow. Infrastructure elements like tunnels, bridges, roundabouts, and toll booths are all represented. This kind of diversity is exactly what end-to-end driving models need to handle real-world complexity. Whether you're working on neural reconstruction, synthetic data generation, scenario mining, or direct end-to-end autonomous driving systems, this dataset provides the scale and variety to push research forward. The multi-sensor fusion aspect is particularly important. With synchronized camera, LiDAR, and radar data, along with ego motion and calibration information, researchers can explore how different sensor modalities complement each other in challenging conditions where single sensors might fail. NVIDIA is also releasing a developer kit to help navigate this massive dataset efficiently, so you can download only the specific sensors, geographic regions, or environmental conditions relevant to your research. This is the kind of open resource that accelerates the entire field. Accessible at Hugging Face under NVIDIA's Autonomous Vehicle Dataset License Agreement. #physicalai #deeplearning #computervision #neuralnetworks #opensource