TLDR: The Valeo Near-Field (VNF) dataset is a new multi-modal dataset designed to improve pedestrian intent detection around stationary vehicles. It includes synchronized data from fisheye cameras, LiDAR, ultrasonic sensors, and motion capture, offering detailed 3D body pose and position annotations in diverse real-world scenarios. This dataset addresses limitations of existing resources by focusing on near-field interactions and provides a benchmark for developing advanced pedestrian perception and safety algorithms for intelligent vehicles.
As intelligent vehicles become more common, ensuring their safe and effective interaction with pedestrians is a significant challenge. A crucial part of this interaction involves predicting and understanding a pedestrian’s intentions as they approach a parked vehicle. This capability is vital for advanced driver-assistance systems (ADAS), allowing a vehicle to anticipate actions like a pedestrian wanting to open a door or access the trunk.
Current datasets for pedestrian detection and tracking often fall short in addressing the complexities of real-world dynamic interactions between pedestrians and vehicles, especially in the immediate vicinity of a stationary car. Many existing datasets focus on pedestrians at crossings or lack the necessary multi-modal sensor integration to capture intricate pedestrian behaviors in urban settings. The absence of high-quality, synchronized multi-modal data for relevant intention scenarios has hindered the development of robust algorithms for pedestrian detection and intent prediction in the near-field of vehicles.
Introducing the Valeo Near-Field (VNF) Dataset
To tackle these challenges, researchers from Valeo, BRAIN Division, and Université Paris-Saclay have introduced a novel dataset called Valeo Near-Field (VNF). This dataset is specifically designed to study pedestrian interactions with a stationary vehicle in a near-field perception setting. It comprises 300 sequences of synchronized multi-modal data, including feeds from fisheye cameras, LiDAR laser scans, ultrasonic sensor readings, and motion capture-based 3D body poses. These sequences, each lasting about 1 minute and captured at 30 frames per second, were collected across diverse real-world scenarios involving 13 participants.
A key contribution of the VNF dataset is its detailed annotations of 3D body joint positions, precisely synchronized with fisheye camera images. Additionally, accurate 3D pedestrian positions are extracted from LiDAR data, providing a robust foundation for benchmarking perception algorithms. The researchers have made a portion of this dataset publicly available, along with a comprehensive benchmark suite that includes evaluation metrics for accuracy, efficiency, and scalability on embedded systems.
Addressing Real-World Challenges
The VNF dataset is unique in its focus on real-world challenges such as sensor occlusions, dynamic environments, and hardware constraints. It offers an invaluable resource for developing and evaluating cutting-edge algorithms in pedestrian detection, 3D pose estimation, and 4D trajectory and intention prediction. The dataset also provides baseline performance metrics using custom neural network architectures and suggests future research directions to encourage its adoption and enhancement. This work aims to serve as a foundation for researchers seeking to advance the capabilities of intelligent vehicles in near-field scenarios.
Data Collection and Synchronization
The data collection process involved a BMW G11 S7 vehicle equipped with a comprehensive sensor suite. This included a Surround-view Camera System (SVCS) with fisheye cameras for a 360-degree field of view, four LiDAR sensors for precise distance measurements, and ultrasonic sensors for close-range proximity detection. A MVN Awinda motion capture suit was used to capture the 3D body poses of pedestrians, providing ground truth for 3D body joint positions.
A significant challenge was synchronizing the vehicle’s sensors with the motion capture suit. A tailor-made synchronization technique was developed where participants performed specific arm movements (raising and lowering left, then right, then both arms) while in a stationary pose. These movements were automatically matched across image and motion suit data to establish time synchronization. To mitigate positional drift from the motion suit, pedestrian 3D positions and orientations were manually annotated using LiDAR scans at 3 frames per second, and the motion suit data was aligned accordingly.
Experiments and Baseline Results
The paper describes experiments using state-of-the-art methods for 3D keypoint-based applications. This involved using YOLOX for pedestrian detection, a fisheye geometry-aware version of ViTPose for 2D skeleton estimation, and a method for uplifting 2D keypoints to 3D pose estimation. Pedestrian localization was further refined using a transformer-based approach that accounts for multi-view fisheye camera geometry.
Performance was evaluated across different detection zones (0-5m, 5-10m, 10-15m, 15-20m) using metrics like Mean Per Joint Position Error (MPJPE) for 3D pose detection, and accuracy and Average Displacement Error (ADE) for pedestrian localization. The results showed consistent ADE across zones when pedestrians were correctly matched, but accuracy was higher in the 5-10m zone, possibly due to better visibility from multiple cameras and fewer occlusions from vehicle doors. MPJPE generally increased with distance, except for the very close range where some body parts might be out of the camera’s field of view. The dataset also showed similar performance for male and female participants and in indoor/outdoor settings.
Also Read:
- Mapping Emotions at Work: Insights from a 30-Month Study
- Advancing Parkinson’s Assessment with Relaxed Multimodal AI
Future Directions
While the VNF dataset makes significant contributions, it has limitations, primarily focusing on stationary vehicle interactions and not extensively covering diverse weather conditions. Future research can enhance individual perception tasks, explore end-to-end processing pipelines, and leverage synthetic data augmentation or self-supervised learning. The researchers hope this dataset will significantly advance pedestrian perception and safety in autonomous driving systems. You can find more details about this work in the full research paper: Valeo Near–Field: a novel dataset for pedestrian intent detection.
