TLDR: MoSE is a new AI model for autonomous driving that mimics human learning by breaking down driving into skills and reasoning step-by-step. It uses a Mixture-of-Experts approach with a skill-oriented routing mechanism, allowing it to achieve state-of-the-art performance on complex driving tasks with significantly fewer active parameters than larger models, making it more efficient and interpretable.
A new approach to developing AI for autonomous driving, inspired by how human drivers learn, has been introduced. This method, called MoSE (Mixture-of-Skill-Experts), aims to make self-driving systems more generalized and easier to understand, while also being computationally efficient.
Traditional large language models (LLMs) and vision language models (VLMs) used in autonomous driving often require vast amounts of training data and complex optimization. MoSE addresses these challenges by mimicking the human learning process: skill-by-skill and step-by-step. It uses a unique skill-oriented routing mechanism that defines and annotates specific driving skills. This allows different “experts” within the model to specialize in various scenarios and reasoning tasks, leading to more focused and efficient learning.
The researchers also aligned the driving process with multi-step planning, similar to human reasoning. They built a hierarchical skill dataset and pre-trained the model’s router to encourage it to “think” step-by-step. This means the model can integrate various auxiliary tasks like description, reasoning, and planning into a single forward process without adding extra computational cost.
One of the significant achievements of MoSE is its efficiency. With less than 3 billion sparsely activated parameters, it outperforms several models with 8 billion or more parameters on the CODA autonomous driving corner case reasoning task. This represents a substantial reduction in activated model size, at least by 62.5%, while achieving state-of-the-art performance in single-turn conversations.
The skill-centric routing mechanism is key to MoSE’s success. It allows the model to understand the driving scene and input text more precisely, selecting the right experts for each stage of the driving context. This leads to a structured chain of activated skills across different hierarchical levels, which not only aids the model’s reasoning and training but also provides better interpretability during operation. For instance, the model might first detect objects, then predict their behaviors, and finally evaluate their importance for driving decisions.
Experiments on the CODA dataset, which focuses on multi-modal corner cases in driving, showed MoSE’s superior performance across general perception, regional perception, and driving suggestions tasks. The model also demonstrated better performance scaling with increasing data sizes compared to general Mixture-of-Experts models, indicating its potential for even larger and more complex datasets. Furthermore, MoSE proved its adaptability by extending its effectiveness to the DriveLM dataset, which covers more common scenarios and focuses on driving planning and trajectory estimation.
Also Read:
- LeAD: A New Autonomous Driving System Combines Real-time Control with Advanced AI Reasoning
- GO-Skill: A New AI Approach for Robots to Learn Complex Tasks by Abstracting Reusable Skills
The development of MoSE marks a promising direction for future autonomous driving systems, offering a balance between model complexity, training efficiency, and data requirements. For more technical details, you can refer to the original research paper: MoSE: Skill-by-Skill Mixture-of-Expert Learning for Autonomous Driving.
