TLDR: This research introduces a five-layer model for representing driving scenarios, enabling the generation and evaluation of rare and challenging ‘Edge Cases’ for autonomous vehicles. By using large foundational models (LLMs and MLLMs) and a data augmentation strategy, the method creates diverse synthetic scenarios from real-world data. New metrics, diversity and originality scores, are proposed to assess the relevance of these generated scenarios. The approach aims to improve autonomous vehicle development by providing a structured, controllable, and evaluable way to simulate critical driving situations.
Autonomous vehicles rely heavily on the ability to handle rare and challenging driving situations, often called ‘Edge Cases’. Since these scenarios are difficult to encounter in real-world driving, researchers frequently turn to simulations and generative models to create synthetic versions. A new research paper, “Driving scenario generation and evaluation using a structured layer representation and foundational models,” by Arthur Hubert, Gamal Elghazaly, and Raphaël Frank, introduces an innovative approach to address this challenge.
The paper proposes a structured five-layer model to represent driving scenarios. This model breaks down a complex driving situation into five distinct components: road structures, surrounding structures, temporary changes to these structures, dynamic objects (like other vehicles or pedestrians), and environmental conditions. This structured approach allows for a more precise and controllable way to define and modify scenarios, which is crucial for generating diverse and relevant edge cases.
Unlike previous methods, this new structure introduces subclasses and specific characteristics for every agent within a scenario. This detailed representation enables a more nuanced comparison between different scenarios using a unique embedding system tailored to their layer model. The researchers leverage large foundational models, specifically Large Language Models (LLMs) and Multimodal Large Language Models (MLLMs), to generate new driving scenarios. Their strategy involves augmenting existing real-world datasets by selectively editing specific components of a scenario, ensuring that the generated scenes remain coherent and grounded in reality.
A key aspect of this research is the development of new metrics to evaluate the quality of these synthetic datasets. The ‘diversity score’ measures how different the generated scenarios are from each other, while the ‘originality score’ assesses how similar a synthetic dataset is to a real reference set. These metrics are crucial for determining if the generated scenarios truly represent novel and challenging edge cases. The evaluation can be performed layer by layer, providing a deeper understanding of the scene’s originality and diversity.
The team used the nuScenes dataset and the Gemini 2.5 Pro MLLM to acquire and describe real reference scenes. They then used Gemini 2.5 Pro as the LLM for generating new scenarios and Gemini-embedding-001 for calculating semantic similarities. Their experiments showed that providing more structure and guidance to the LLM generally improved the generation process, although overly rigid structures could sometimes hinder performance.
The paper also includes a qualitative evaluation of synthetic videos generated from these structured scenario descriptions using the Veo3-preview World Foundation Model. While image-guided generation showed better fidelity to the original scenes, it sometimes struggled with inconsistencies when edited elements were not present in the initial frame. Text-guided generation, on the other hand, could sometimes lead to more ‘hallucinations’ but offered greater flexibility.
Also Read:
- Modeling Realistic Pedestrian-Driver Interactions with Human-Like Constraints
- Cognition Envelopes: Guarding AI Decisions in Autonomous Search and Rescue
In conclusion, this research presents a significant step forward in generating diverse and relevant driving scenarios for autonomous vehicle development. By combining a structured five-layer representation with the power of foundational models and tailored evaluation metrics, the authors provide a flexible and effective method for expanding the operational domain of autonomous systems. For more details, you can read the full paper here.
