TLDR: SynAD is a new framework that improves real-world End-to-End Autonomous Driving (E2E AD) models by integrating synthetic driving data. Traditionally, synthetic data was hard to use because it lacked sensor inputs like cameras or LiDAR. SynAD solves this by generating ego-centric synthetic scenarios and using a “Map-to-BEV Network” to create Bird’s-Eye-View features directly from maps. This allows synthetic data to be effectively combined with real data during training, significantly enhancing safety performance and creating more robust autonomous driving systems.
Autonomous vehicles are rapidly transitioning from research labs to public roads, largely thanks to breakthroughs in deep learning and the availability of extensive real-world driving datasets. These advancements have significantly propelled the development of end-to-end autonomous driving (E2E AD) models, which integrate perception, prediction, and planning into a single, unified system.
However, relying solely on real-world data presents a significant challenge: it’s expensive to collect and label, leading to a limited diversity of driving scenarios for training. This means that many rare or safety-critical situations are underrepresented, making it difficult to train robust self-driving systems. Synthetic scenario generation has emerged as a promising solution to enrich training data by creating a wider variety of driving situations.
The problem is that integrating these synthetic scenarios into real-world E2E AD models has been largely unexplored. Current synthetic scenario generation methods typically produce only path-level outputs and don’t include the necessary sensor inputs, such as multi-camera images or LiDAR data, that real-world E2E AD models depend on. This absence of an ‘ego vehicle’ (the self-driving car) and its associated sensor data has been a major barrier.
To address this critical gap, researchers have introduced SynAD, the first framework designed to enhance real-world E2E AD models using synthetic data. SynAD tackles the challenge of integrating synthetic scenarios by focusing on three key components.
Ego-centric Scenario Generation
SynAD first generates realistic multi-agent driving scenarios that can be tailored to specific conditions. Unlike previous methods, it then designates one agent as the ‘ego vehicle’ – the self-driving car – based on which vehicle has the most comprehensive driving information (e.g., traveling the longest distance). The paths of this ego vehicle and other agents are then converted into an ego-centric map representation. This means the map is centered around the ego vehicle, with its driving direction aligned to a standard axis, and includes bounding boxes for other vehicles. This process effectively transforms abstract path data into a format usable by autonomous driving systems.
Map-to-BEV Network
A crucial innovation in SynAD is the Map-to-BEV Network. This component solves the problem of missing sensor inputs in synthetic data. It generates Bird’s-Eye-View (BEV) features directly from the ego-centric map inputs, without needing camera images or LiDAR data. To ensure these map-derived BEV features are consistent with those from real-world sensors, the network is trained to align its outputs with BEV features extracted from actual multi-camera images of real scenarios. This alignment allows the E2E AD model to process synthetic map data as if it were receiving sensor data.
Also Read:
- A Unified Approach to Autonomous Vehicle Motion Planning with Multi-Dataset Learning
- Autonomous Vehicles Gain Self-Awareness: Predicting the Road Ahead with Quantified Uncertainty
Effective Training Strategy
Finally, SynAD devises a smart training strategy to integrate these map-based synthetic data with real driving data. The framework incorporates map-based BEV features into the motion forecasting and planning modules of the E2E AD model. These modules benefit significantly from additional contextual information like road geometry and traffic structure, which maps readily provide. Interestingly, the occupancy prediction module, which requires high spatial precision, does not use map inputs to avoid performance degradation, demonstrating a selective and optimized integration approach.
Experimental results demonstrate that SynAD effectively integrates all its components and notably enhances safety performance. While the model might show slightly higher L2 distance errors in trajectory prediction due to the broader distribution of behaviors in synthetic scenarios, it achieves a significantly lower collision rate compared to existing methods. This trade-off is highly beneficial for real-world autonomous driving, where collision avoidance is paramount. SynAD also shows robust performance in motion forecasting and occupancy prediction, even when synthetic data is introduced as a new input type.
By bridging the gap between synthetic scenario generation and real-world E2E AD, SynAD paves the way for more comprehensive, robust, and safer autonomous driving models. This framework allows developers to leverage the vast potential of synthetic data to train self-driving cars for a wider array of complex and critical situations that are difficult or dangerous to encounter in real-world testing.
