TLDR: A new research paradigm called E3AD integrates human driving cognition, derived from EEG signals and a large brain model (LaBraM), into end-to-end autonomous driving systems. By using a two-stage contrastive learning process, E3AD significantly improves planning performance and reduces collision rates in baseline models, offering a novel approach for safer and more robust autonomous vehicles.
Autonomous driving technology is constantly evolving, aiming to transfer control from human drivers to advanced AI systems for improved efficiency and safety. A new research paradigm, E3AD (Embodied Cognition Augmented End2End Autonomous Driving), introduces a novel approach to enhance these systems by integrating human driving cognition.
Traditional end-to-end autonomous driving systems often rely on visual feature extraction networks that are trained using labeled data. However, this method can limit the model’s ability to generalize and adapt to diverse driving scenarios. The human brain, in contrast, uses a more holistic “embodied reasoning” to anticipate dangers and adjust to new situations.
The E3AD paradigm proposes a comparative learning method. It involves training visual feature extraction networks by contrasting them with a general EEG (Electroencephalography) large model, specifically LaBraM. LaBraM is a powerful brain-inspired model capable of extracting rich cognitive features directly from EEG signals. By leveraging these cognitive features, E3AD aims to provide a broader form of supervision to the visual feature networks, allowing them to learn latent human driving cognition.
How E3AD Works
The researchers collected a unique cognitive dataset that pairs video data with corresponding EEG segments. This dataset was crucial for the contrastive learning process. The training is divided into two main stages:
1. Driving-Thinking Model Training: A visual feature extraction network, called the “Driving-Thinking Model,” is trained using the self-collected dataset. This model learns to infer driving-related cognitive information from visual inputs by contrasting its outputs with the cognitive features extracted by the frozen LaBraM model. This is inspired by the successful CLIP paradigm for cross-modal learning.
2. Embodied Cognition Augmented End2End Model Training: After the Driving-Thinking Model is trained and its parameters are frozen, it is integrated into popular end-to-end autonomous driving frameworks. The enhanced model is then trained on large-scale public autonomous driving datasets, such as nuScenes, without any further EEG input. This ensures fairness and consistency with other baseline models.
The study explored three different frameworks for integrating the learned driving cognition into end-to-end planning:
- Attaching to Spatio-temporal features: This framework hypothesizes that the Driving-Thinking model learns human attention mechanisms towards visual features, helping to select relevant visual information for planning.
- Interacting with the Ego Query: Here, the hypothesis is that the Driving-Thinking model acquires driving-related cognitive knowledge that can guide how the ego vehicle’s historical motion information interacts with visual features to generate planning features.
- Interacting with Planning Features: This framework suggests that the Driving-Thinking model learns more advanced, planning-specific cognition, enabling it to directly influence and improve preliminary planning results. This framework showed the most significant improvements in performance.
Also Read:
- Modeling Realistic Pedestrian-Driver Interactions with Human-Like Constraints
- Enhancing Autonomous Driving with Structured Scenario Generation and Foundational Models
Significant Improvements in Driving Performance
Experiments conducted on public datasets like nuScenes and in the CARLA simulation environment (using Bench2Drive) demonstrated that the E3AD paradigm significantly boosts the planning performance of baseline autonomous driving models. For instance, it led to a notable reduction in collision rates for models like UniAD and VAD-Base, often surpassing state-of-the-art methods. The method also showed improvements in driving scores and route completion rates in closed-loop simulations, indicating enhanced safety and robustness in challenging scenarios.
Ablation studies confirmed that the performance gains were indeed due to the contrastive learning process with LaBraM, rather than just additional visual features or the capacity of the video encoder. Furthermore, using EEG data from expert drivers yielded better results, and mixing data from both expert and novice drivers further improved performance, suggesting the value of diverse cognitive data.
This research marks a significant step as the first work to integrate human driving cognition to improve end-to-end autonomous driving planning. It represents an initial effort to incorporate embodied cognitive data into autonomous driving, offering valuable insights for future brain-inspired AI systems. The researchers plan to make their code available at https://github.com/AIR-DISCOVER/E-cubed-AD.
