TLDR: KEPT (Knowledge-Enhanced Prediction of Trajectories) is a new framework that significantly improves trajectory prediction for autonomous driving. It uses a specialized video encoder (TFSF) to capture motion dynamics, a fast retrieval system (k-means + HNSW) to find similar past driving scenarios, and a three-stage fine-tuning process for Vision-Language Models (VLMs) to enhance spatial perception and motion reasoning. By employing Chain-of-Thought prompting with retrieved examples, KEPT achieves state-of-the-art accuracy and safety on the nuScenes dataset, offering a more interpretable and deployable solution for predicting vehicle paths.
Autonomous driving systems rely heavily on accurately predicting the paths of vehicles to ensure safety and efficiency on the road. While advanced vision-language models (VLMs) have shown great promise, they often struggle to truly understand the dynamic nature of a scene and incorporate real-world driving knowledge into their predictions. This challenge is particularly evident in complex or unusual scenarios where errors can quickly accumulate.
To address these critical limitations, a new framework called KEPT (Knowledge-Enhanced Prediction of Trajectories) has been introduced. This innovative system aims to improve how VLMs predict vehicle trajectories by directly analyzing consecutive front-view driving frames. KEPT integrates several key components to achieve its enhanced performance, offering a more interpretable and reliable approach to autonomous driving.
Understanding KEPT’s Core Components
At the heart of KEPT is a sophisticated **Temporal Frequency-Spatial Fusion (TFSF) video encoder**. This encoder is designed to process video clips, specifically seven frames captured over three seconds, to create robust representations that capture important motion dynamics. It achieves this by combining frequency-domain attention (using Fast Fourier Transform) with multi-scale spatial features and a temporal transformer. The encoder is trained using a self-supervised learning method that includes ‘hard-negative mining,’ which helps it learn to distinguish between similar but distinct driving situations effectively.
Once these video clips are encoded, they are stored in a **scalable embedding-retrieval pipeline**. This pipeline acts as a knowledge base, allowing the system to quickly find similar past driving scenarios. It uses a two-stage approach: first, k-means clustering groups similar scenes, and then a Hierarchical Navigable Small World (HNSW) graph efficiently searches within these clusters. This setup ensures that when a new driving situation arises, KEPT can rapidly retrieve relevant historical examples and their corresponding ground-truth trajectories, providing valuable context for prediction.
A crucial aspect of KEPT is its **triple-stage fine-tuning schedule** for Vision-Language Models. This incremental training process ensures the VLM’s language understanding capabilities are aligned with the specific demands of trajectory prediction:
- **Stage A: Spatial Perception Enhancement:** The VLM is trained to improve its understanding of spatial relationships, such as object categories, sizes, and distances from monocular camera inputs. This is done using a visual question-answering (VQA) format.
- **Stage B: Surround-View Trajectory Regression:** The model learns to predict ego-centric trajectories from six synchronized cameras and basic vehicle kinematics. This stage focuses on generating physically realistic motions, ensuring predictions are smooth and avoid collisions.
- **Stage C: End-to-End Front-View Trajectory Prediction:** In the final stage, KEPT is fine-tuned to predict trajectories directly from a sequence of seven front-view frames, integrating the spatial perception and motion reasoning learned in previous stages.
Finally, KEPT employs a **Chain-of-Thought (CoT) prompting paradigm**. Instead of directly asking for a prediction, the VLM is presented with multiple retrieved reference scenes (images, kinematics, waypoints, and future trajectories) before the target scene. It is explicitly instructed to compare these examples and reason under specific planning constraints, such as collision avoidance, speed smoothness, and staying within drivable areas. This approach makes the prediction process more transparent and controllable, mimicking human-like decision-making.
Performance and Practicality
Evaluated on the nuScenes dataset, KEPT demonstrates state-of-the-art performance across various open-loop protocols. For instance, under the NoAvg protocol, it achieved an impressive 0.70m average L2 error with a mere 0.21% collision rate. When provided with lightweight ego status information, its performance further improved to 0.31m average L2 and a 0.07% collision rate, outperforming several existing methods like UniAD, VAD-Base, and Drive-OccWorld.
Ablation studies confirmed the complementary benefits of all three fine-tuning stages, showing that removing any stage significantly degrades long-horizon stability and safety. The studies also revealed that using the Top-2 retrieved exemplars yielded the best balance between accuracy and safety, providing just enough informative priors without introducing distracting noise. Furthermore, the k-means-clustered HNSW index achieved sub-millisecond retrieval latency, making KEPT suitable for practical, real-time deployment.
Also Read:
- SAM-LLM: A New Approach to Interpretable Lane Change Prediction for Autonomous Vehicles
- Adapting Large Language Model Techniques for Autonomous Driving Motion Generation
Looking Ahead
While KEPT represents a significant step forward, the researchers acknowledge certain limitations. The current retrieval priors are based on nuScenes-like distributions, which might limit performance in highly unusual or out-of-domain scenarios. Future work aims to extend KEPT to multi-sensor inputs, improve cross-city generalization, and develop active, safety-aware retrieval mechanisms that can update the knowledge base on-the-fly. The lightweight nature of the Qwen-series VLMs used as the backbone also makes KEPT a promising foundation for trustworthy and scalable VLM planning in autonomous driving.
For more technical details, the full research paper can be accessed here.
