TLDR: The SoM-MIMO system is a novel approach for efficiently and robustly transmitting images for cooperative AI tasks, like instance segmentation, in networked mobile agents. Inspired by human synesthesia, it integrates image features with digital MIMO communication systems. By using hierarchical feature fusion and channel-aware encoding, SoM-MIMO significantly improves perception accuracy, especially in challenging signal conditions, while maintaining or reducing communication overhead compared to traditional methods.
In an increasingly connected world, where autonomous vehicles, drones, and robots interact in real-time, the ability to efficiently and reliably transmit sensory data, especially images, is paramount. This capability, known as cooperative perception (CP), allows multiple mobile agents to share information, leading to more accurate environmental understanding and better decision-making. However, current methods for transmitting image data, even those using advanced deep learning techniques combined with multiple-input multiple-output (MIMO) communication systems, often fall short when it comes to complex tasks and real-world digital communication environments.
Traditional approaches to image transmission often separate the process of compressing the image (source coding) from protecting it during transmission (channel coding). While deep learning-based joint source-channel coding (JSCC) has shown promise, existing solutions for MIMO systems are typically designed for simpler image reconstruction tasks and don’t fully integrate with modern digital MIMO communication systems. This gap limits their effectiveness in supporting the sophisticated perception tasks required by networked mobile agents.
Inspired by the concept of human synesthesia, where one sense automatically evokes another to perform cognitive tasks, researchers have proposed the Synesthesia of Machines (SoM) framework. This framework aims to intelligently integrate multi-modal information from various ‘machine senses,’ such as communication devices and sensors, to enhance intelligent functionalities. Building on this principle, a new system called SoM-MIMO has been developed, which is a task-driven MIMO system specifically designed for image transmission.
The SoM-MIMO system addresses the limitations of previous methods by deeply coupling the digital MIMO communication system with the specific perceptual tasks of mobile agents. It leverages two key properties: the structural characteristics of feature pyramids (which are crucial for perceptual tasks like object detection and instance segmentation) and the channel properties of closed-loop MIMO communication systems. This integration allows for highly efficient and robust digital MIMO transmission of images.
At its core, SoM-MIMO employs a Hierarchical Feature Fusion (HFF) module. When an image is captured, it’s first processed to extract a feature pyramid, which contains rich multi-scale information but is also very large. The HFF module intelligently fuses these features, significantly reducing data redundancy and compressing the data volume by over 95% while preserving critical information. This makes the data much more manageable for transmission.
Following this, an MIMO Channel-aware Encoding (MCE) module comes into play. This module is designed to make the feature transmission robust against varying channel conditions. It uses physical layer information, such as the Signal-to-Noise Ratio (SNR) and Channel State Information (CSI), to adaptively distribute features across different data streams. This means that more important features can be sent over higher-quality sub-channels, protecting them from distortion.
Finally, the system incorporates standard MIMO baseband processing. This includes a fully connected layer for further channel compression, a non-linear activation function (like tanh) to prepare features for quantization and improve training stability, and then quantization, modulation, and SVD precoding. These steps ensure that the SoM-MIMO system is fully compatible with existing digital communication infrastructures.
The effectiveness of SoM-MIMO was rigorously tested using the challenging instance segmentation task in complex urban traffic scenarios. Compared to two baseline JSCC schemes, SoM-MIMO achieved significant improvements in mean Average Precision (mAP) – 6.30 and 10.48 average mAP improvements across all SNR levels, with even more dramatic gains (15.87 and 16.83 mAP) at very low SNR conditions, all while maintaining the same communication overhead. Furthermore, it demonstrated the ability to achieve comparable performance with substantially reduced transmission overhead, offering flexibility in balancing communication costs and perception quality.
Ablation studies confirmed the critical role of the MIMO channel-aware encoding and decoding modules in adapting to diverse SNR conditions. The system also showed strong adaptability to different numbers of antennas and quantization precision levels, highlighting its versatility for practical MIMO digital communication systems. For more technical details, you can refer to the full research paper here.
Also Read:
- VariAntNet: Deep Learning for Faster, Decentralized Swarm Robot Control
- AppCopilot: Advancing Mobile AI Agents for Everyday Use
This research marks a significant step towards enabling more reliable and efficient cooperative perception for networked mobile agents, paving the way for advanced applications in autonomous driving, smart cities, and robotics. Future work will explore multi-user communication designs and intelligent fusion mechanisms for even greater performance gains.
