TLDR: MMEdge is a novel framework designed to accelerate real-time multimodal inference on resource-constrained edge devices. It achieves this by decomposing the inference process into fine-grained, pipelined sensing and encoding units, allowing incremental computation as data arrives. To maintain accuracy, it incorporates a lightweight temporal aggregation module. Furthermore, MMEdge features an adaptive multimodal configuration optimizer that dynamically selects optimal sensor and model settings, and a cross-modal speculative skipping mechanism to bypass slower modalities when early predictions are confident. Evaluations on public datasets and a real-world UAV testbed demonstrate that MMEdge significantly reduces end-to-end latency while preserving high task accuracy under dynamic conditions.
Real-time artificial intelligence (AI) that combines information from multiple types of sensors, known as multimodal inference, is becoming crucial for applications like autonomous driving, human-computer interaction, and mobile health. However, performing these complex tasks on small, resource-limited devices like smartphones or drones presents significant challenges. Traditional methods often struggle with delays because they wait for all sensor data to arrive before processing, and they don’t efficiently handle the different speeds at which various sensors provide data.
A new research paper introduces a framework called MMEdge, designed to accelerate on-device multimodal inference. The paper, titled “MMEdge: Accelerating On-device Multimodal Inference via Pipelined Sensing and Encoding,” was authored by Runxi Huang, Mingxuan Yu, Mingyu Tsoi, and Xiaomin Ouyang from the Hong Kong University of Science and Technology. You can read the full paper here: MMEdge Research Paper.
The Core Idea: Pipelined Processing
MMEdge tackles the latency problem by rethinking how sensor data is processed. Instead of waiting for a complete set of sensor inputs, it breaks down the entire inference process into smaller, fine-grained “sensing and encoding units.” This allows computation to begin incrementally as data arrives, creating a pipeline where sensing and encoding happen in parallel. This pipelined approach eliminates idle waiting times and significantly reduces the overall time it takes to get a prediction.
Maintaining Accuracy with Temporal Aggregation
While pipelining speeds things up, processing data in smaller chunks can sometimes lead to a loss of temporal context, potentially affecting accuracy. MMEdge addresses this with a lightweight “temporal aggregation module.” This module is designed to capture rich temporal dynamics across different pipelined units. It uses techniques like alternating temporal shifts and extracting multi-scale temporal difference features to ensure that important temporal and semantic relationships are maintained, even when data is processed in a fine-grained manner.
Adapting to Dynamic Conditions
Edge devices operate in varied and often unpredictable environments, where resource availability and data complexity can change rapidly. To handle this, MMEdge includes two key adaptive mechanisms:
- Adaptive Multimodal Configuration Optimizer: This component dynamically selects the best sensing and model configurations for each sensor modality. It considers factors like frame rates, audio chunk sizes, and model complexities (e.g., using a smaller or larger neural network) to meet latency constraints while maximizing accuracy. It uses an offline-trained accuracy predictor and latency profiles to make these decisions efficiently in real-time.
- Cross-Modal Speculative Skipping: Multimodal systems often face situations where some sensors (e.g., audio) provide data much faster than others (e.g., video). This can cause slower modalities to hold up the entire process. MMEdge introduces a speculative skipping mechanism that allows the system to bypass future units of slower modalities if an early prediction, based on available data, reaches a sufficient level of confidence. This reduces unnecessary computation and waiting time without compromising accuracy.
Also Read:
- Accelerating Edge AI Deployment for Industrial Operations
- Keeping VLA Models Sharp: Aligning Visual Representations for Better OOD Performance
Real-World Evaluation and Results
The researchers evaluated MMEdge using two public multimodal datasets (Lip Reading in the Wild and NuScenes-Mini-QA) and deployed it on a real-world unmanned aerial vehicle (UAV) testbed for human tracking. The UAV testbed provided a challenging environment with dynamic conditions, including unstable voltage, in-flight vibrations, and varying distances to targets, which can affect sensor quality and resource availability.
The results were compelling: MMEdge significantly reduced end-to-end latency while maintaining high task accuracy across various system and data dynamics. For instance, on the UAV dataset, MMEdge achieved a mean Intersection over Union (IoU) of 70.47%, close to the highest-accuracy baseline, but reduced end-to-end latency by over 80%. On the LRW dataset, it reduced latency to around 137 ms, the lowest among all methods, with less than a 3% accuracy compromise compared to the best-performing baseline.
The ablation studies further confirmed the importance of each component: removing the temporal aggregation module led to a significant accuracy drop, while disabling speculative skipping or adaptive configuration increased latency. MMEdge’s ability to adapt to different latency budgets, varying resource availability, and diverse input sample complexities demonstrates its robustness and effectiveness for real-time multimodal AI on resource-constrained edge devices.
