TLDR: EMC2 (Edge-based Mixture of Experts Collaborative Computing) is a new system for autonomous vehicles that achieves both high accuracy and low latency in 3D object detection. It uses a ‘Mixture of Experts’ architecture, dynamically selecting specialized sub-models based on driving scenarios (object distance and clarity). By fusing LiDAR and camera data and incorporating extensive hardware-software optimizations, EMC2 significantly outperforms existing methods in accuracy and inference speed on edge platforms like Jetson, making real-time, reliable autonomous driving perception more feasible.
Autonomous vehicles (AVs) rely heavily on their perception systems to understand their surroundings, acting as the ‘eyes’ of the vehicle. A critical function of these systems is 3D object detection, which involves identifying and locating objects in the vehicle’s environment. For safe and reliable autonomous driving, this detection must be both highly accurate and incredibly fast. However, achieving both simultaneously has been a significant challenge due to limited computing resources and the complex, dynamic nature of real-world driving scenarios.
Traditional approaches often face a trade-off: systems designed for high accuracy tend to be slow, while fast systems might compromise on precision. This dilemma is particularly pronounced when deploying these systems on ‘edge’ platforms, such as the embedded computers found in AVs, which have limited processing power and memory.
Introducing EMC2: A Smart Solution for 3D Object Detection
Researchers have developed a novel computing system called Edge-based Mixture of Experts Collaborative Computing (EMC2) to address this challenge. EMC2 is designed specifically for autonomous vehicles, aiming to deliver both low-latency (fast) and high-accuracy 3D object detection. Unlike conventional systems that use a single, complex model for all scenarios, EMC2 employs a ‘Mixture of Experts’ (MoE) architecture. This means it has multiple specialized sub-models, or ‘experts,’ and dynamically chooses the most suitable one based on the current driving situation.
The system effectively combines data from different sensors, primarily LiDAR (which provides sparse 3D point clouds) and cameras (which offer dense 2D images). By fusing these complementary data sources, EMC2 creates a more robust understanding of the environment.
How EMC2 Works: Key Components
EMC2 operates through several integrated components:
Adaptive Multimodal Data Bridge (AMDB): This is the initial processing unit that takes raw data from LiDAR and cameras. It preprocesses this multimodal input, extracting features and generating ‘proposal regions’—potential areas where objects might be located. Depending on the scenario, it can extract detailed image features and project them into 3D space, fusing them with LiDAR data to create rich, multimodal representations for more complex detection tasks.
Scenario-Adaptive Dispatcher (SAD): This is the ‘brain’ of the MoE system. Based on two key parameters—object distance and clarity (inferred from the confidence of the proposal regions)—the SAD dynamically routes the processed data to the most appropriate expert. For instance, if objects are close and clearly visible, it dispatches the task to a simpler, faster expert. If objects are distant or unclear, it sends the task to a more complex, accurate expert that can leverage more data.
Scenario-Optimized Experts: EMC2 features three specialized experts:
- Latency-Prioritized Expert (LPE): Designed for simple scenarios with close and distinct objects. It uses 2D processing, which is computationally less expensive, to achieve very fast detection.
- Versatile Efficiency Expert (VEE): Handles mixed visibility cases, where some objects might be distant but clear, or near but unclear. It uses 3D processing to maintain accuracy when LiDAR data is less complete.
- Accuracy-Prioritized Expert (APE): Activated for the most challenging scenarios, such as distant and unclear objects. This expert integrates both LiDAR and camera data to compensate for missing information, ensuring high precision even in difficult conditions.
Optimizing for Performance on Edge Devices
To ensure EMC2 runs efficiently on resource-constrained edge devices like the NVIDIA Jetson platforms, the researchers implemented several hardware-software optimizations:
Collaborative Training: A hierarchical training strategy with triple back-propagation helps stabilize the learning process for the experts, especially when initial data proposals are unreliable. It also addresses the ‘long-tail effect’ in MoE training, where some experts might receive insufficient data, by using balanced sampling and an adaptive optimizer.
Algorithm Edge-Adaptivity: The system includes a customized 3D sparse convolution library and a Multiscale Pooling technique. Sparse convolution significantly reduces redundant computations by only processing non-empty data points, while Multiscale Pooling adaptively pools image features to manage memory limitations on edge devices.
System-Level Optimization: Further enhancements include memory optimization techniques like overlapping communication and computation, staged thread management, and prefix-sum acceleration for sparse convolutions. Computational graph optimizations, such as model pruning, quantization, and operator fusion, also contribute to improved execution efficiency.
Impressive Results on Benchmarks
Experiments conducted on widely recognized datasets, KITTI and nuScenes, demonstrate EMC2’s superior performance. On the KITTI dataset, EMC2 showed an average accuracy improvement of 3.58% and a remarkable 159.06% inference speedup compared to 15 baseline methods on Jetson platforms. Specifically, it improved pedestrian and cyclist detection accuracy by 5% and 7% respectively in challenging scenarios, and car detection by up to 11% over previous multimodal approaches. For the nuScenes dataset, EMC2 achieved a 3.9% improvement in mean Average Precision (mAP) and a 1.8% boost in NDS (a comprehensive metric) over existing state-of-the-art solutions.
The system’s ability to dynamically switch between experts based on scenario characteristics, combined with its comprehensive algorithmic and system-level optimizations, allows it to achieve a practical balance between accuracy and efficiency. This makes EMC2 highly suitable for real-time deployment in autonomous vehicles. For more technical details, you can refer to the full research paper here.
Also Read:
- Advancing Automated Driving: A Deep Dive into Joint Trajectory Prediction
- LeAD: A New Autonomous Driving System Combines Real-time Control with Advanced AI Reasoning
Future Directions
The researchers plan to continue their work by developing a fully adaptive expert selection mechanism, which could further enhance the system’s adaptability and efficiency for edge deployment in the future.
