TLDR: A new framework enables realistic and controllable editing of pedestrians in multi-view driving videos, allowing for insertion, replacement, and removal. This technology, which uses video inpainting and motion control, helps create synthetic data to improve the robustness of pedestrian detection models in autonomous driving systems, especially for rare but critical scenarios.
Autonomous driving systems rely heavily on robust pedestrian detection models to ensure safety. However, a significant challenge arises from the scarcity of training data for rare but critical pedestrian scenarios, such as jaywalking or pedestrians in close proximity to vehicles. These ‘edge cases’ are crucial for safety but are underrepresented in real-world driving datasets, creating a gap in model performance.
To address this, researchers have developed a novel framework for controllable pedestrian video editing in multi-view driving scenarios. This innovative approach integrates advanced video inpainting techniques with precise human motion control, offering a versatile solution for generating synthetic pedestrian data.
How the Technology Works
The framework begins by identifying pedestrian regions of interest across multiple camera views. These detection bounding boxes are then expanded, resized, and stitched together onto a unified canvas, carefully maintaining the spatial relationships between different camera perspectives. A binary mask is applied to define the editable area, within which pedestrian modifications are guided by a pose sequence control condition. This allows for flexible editing functionalities, including the insertion of new pedestrians, the replacement of existing ones, and even their complete removal from the video.
A key innovation is the ‘dynamic pedestrian region cropping’ strategy. In multi-view driving, pedestrians can appear at various scales – very close or far away. Traditional fixed-size cropping often leads to issues like truncated bodies for close-up pedestrians or a lack of detail for distant ones. This new strategy dynamically crops and resizes pedestrian regions to a uniform scale, simplifying the generation process and ensuring seamless integration across multiple views.
The system also boasts a ‘multi-view consistency framework’. Given the precise calibration of vehicle cameras, 3D pedestrian annotations are projected onto all camera views to create view-specific bounding boxes. These are then processed and organized into a composite image for unified generation, ensuring geometric alignment and semantic consistency across all camera perspectives. For motion control, the framework uses a minimalist skeletal keypoint representation, which is effective for autonomous driving scenarios where fine-grained pose details are less critical than cross-view consistency.
Also Read:
- Enhancing Video Anomaly Detection with AI-Generated Training Data
- Enhancing Human-Object Interaction Understanding with Advanced Amodal Completion
Impact and Applications
Extensive experiments have demonstrated that this framework achieves high-quality pedestrian editing with strong visual realism, spatiotemporal coherence, and cross-view consistency. The capabilities extend beyond simple modifications, allowing for precise control over pedestrian attributes, such as clothing color, through simple text prompts.
Crucially, the research shows that incorporating this synthetically generated pedestrian data into the training of perception models significantly improves their performance. For instance, when tested with the BEVFormer model on the nuScenes dataset, the model trained with synthetic pedestrian samples showed notable improvements in 3D mean Average Precision (mAP) for pedestrian detection. This validates the method’s ability to produce high-fidelity, multi-view consistent pedestrian sequences that effectively enhance perception models’ capability to detect pedestrians in complex driving scenarios.
This method represents a robust and versatile solution for multi-view pedestrian video generation, with broad potential for applications in data augmentation and scenario simulation in autonomous driving. It directly addresses the challenge of data scarcity for safety-critical situations, paving the way for more reliable and safer autonomous vehicles. You can read the full research paper here.
