TLDR: OmniUnet is a new transformer-based neural network designed for planetary rovers to segment unstructured terrain using RGB, depth, and thermal imagery. It helps identify safe traversable areas and obstacles by leveraging complementary sensor data, especially thermal differences in soil types. Tested on a Mars-like dataset, it achieved 80.37% pixel accuracy and demonstrated efficient performance on resource-constrained hardware, making it suitable for on-robot deployment.
Navigating challenging, unstructured environments, such as those found on Mars, presents significant hurdles for robotic systems. To ensure safe and effective exploration, rovers need advanced perception systems that can accurately understand their surroundings. This is where multimodal perception comes into play, combining information from various sensors to overcome the limitations of individual ones.
A new research paper introduces OmniUnet, an innovative neural network architecture designed to enhance terrain segmentation for planetary rovers. This system leverages a combination of RGB (color), depth, and thermal imagery to create detailed maps of the terrain, distinguishing between different surface types and potential obstacles.
The Power of Multimodal Sensing
Traditional perception systems often rely on single sensor types, like standard RGB cameras. However, these can struggle in difficult conditions such as low light, strong glare, or when distinguishing between visually similar surfaces. OmniUnet addresses this by integrating data from three distinct modalities: RGB provides color and texture information, depth sensors offer 3D structural data, and thermal cameras reveal temperature differences. Thermal imagery is particularly valuable for assessing terrain safety, as different soil types exhibit unique thermal behaviors under solar heating. For instance, sandy soils tend to have higher surface temperatures than compacted soils, a distinction that can be crucial for predicting rover slippage, especially on planets like Mars where thermal contrasts are more pronounced.
OmniUnet: A Smart Approach to Terrain Understanding
OmniUnet is built upon a transformer-based architecture, specifically combining the Omnivore backbone for feature extraction with a U-Net-style decoding strategy. This design allows the network to effectively process and integrate heterogeneous data from the different sensors. It starts by converting the input images into ‘patch embeddings,’ which helps the network extract relevant features and correlations across all modalities. The use of a unified architecture means there’s no need for separate models for each sensor type, leading to a more scalable and maintainable system.
The network employs a ‘shifted window attention mechanism’ within its transformer blocks. This allows it to analyze small, non-overlapping sections of the image while also enabling interaction between adjacent regions, ensuring both local detail and global context are considered. During the decoding phase, features from different stages are combined and refined to produce a precise segmentation mask, essentially outlining different terrain types in the rover’s view.
Real-World Testing and Performance
To develop and test OmniUnet, the researchers created a custom multimodal sensor housing using 3D printing. This housing integrated a Realsense D435i stereo camera (for RGB and depth) and an Optris PI-640i thermal camera. This setup was then mounted on the Martian Rover Testbed for Autonomy (MaRTA), a half-scale model of the ExoMars Rosalind Franklin rover.
A new multimodal dataset was collected in the Bardenas semi-desert in northern Spain, an environment chosen for its resemblance to the Martian surface, featuring terrain types like sand, bedrock, and compact soil. A subset of this extensive dataset was meticulously hand-labeled to train the OmniUnet model. The labeled dataset and the software implementation of OmniUnet have been made publicly available to support future research in planetary robotics.
The model’s performance was rigorously evaluated. On the BASEPROD multimodal dataset, OmniUnet achieved a pixel accuracy of 80.37%. It showed strong capabilities in identifying surfaces critical for safe navigation, such as compact terrain (77.44% accuracy) and gravel (51.83% accuracy). While detecting bushes was moderately successful (40.68%), identifying rocks proved more challenging (18.40%), likely due to the varied thermal signatures of different rock types.
Crucially for robotic deployment, OmniUnet demonstrated efficient inference times. When tested on a resource-constrained Jetson Orin Nano computer, it achieved an average prediction time of 673 milliseconds per multimodal image. This confirms its suitability for on-robot deployment where computational resources are limited.
Also Read:
- LightDP: Enabling Real-Time Robot Control on Mobile Devices
- Enhancing Medical Image Translation with Dynamic Test-Time Adaptation
Looking Ahead
The development of OmniUnet marks a significant step forward in enabling more autonomous and safer navigation for planetary rovers. The ability to accurately segment unstructured terrain using a combination of RGB, depth, and thermal data provides rovers with a more comprehensive understanding of their environment. Future work aims to further enhance the model’s accuracy and robustness, especially in distinguishing between a wider range of visually similar terrain classes and improving obstacle detection. Additionally, efforts will focus on designing even more optimized multimodal sensor housings for challenging conditions like low light, further boosting perception reliability for both terrestrial and extraterrestrial applications.
For more technical details, you can refer to the full research paper: OmniUnet: A Multimodal Network for Unstructured Terrain Segmentation on Planetary Rovers Using RGB, Depth, and Thermal Imagery.
