TLDR: This research investigates the impact of using raw, unprocessed satellite sensor data on deep learning models for object detection and classification onboard satellites. The study introduces a simulation workflow to generate raw-like imagery from high-resolution L1 data, allowing for systemic evaluation. It finds that while models trained on raw data can detect objects, they struggle with precise object boundary identification and high-confidence predictions compared to models trained on preprocessed data. The findings highlight the need for adapting AI architectures to better handle raw data complexity for efficient onboard processing.
The frontier of artificial intelligence is rapidly expanding, reaching even into space. A recent study titled “Explaining raw data complexity to improve satellite onboard processing” by Adrien Dorise, Marjorie Bellizzi, Adrien Girard, Benjamin Francesconi, and Stéphane May, delves into the exciting prospect of deploying AI models directly onboard satellites for remote sensing. This advancement promises significant benefits like increased reactivity, better resource management, and bandwidth savings by processing data in orbit rather than on the ground.
Traditionally, AI applications in space have been limited by computational resources, with most processing occurring on Earth using preprocessed Level-1 (L1) images. These L1 images undergo extensive ground-based post-processing, including coregistration, geometric, and radiometric corrections, which are computationally intensive and not ideal for embedded satellite systems. In contrast, raw sensor outputs (referred to as raw products) are noisy, deregistered, and contain various artifacts, posing a unique challenge for AI models.
Bridging the Gap with Simulation
To address the scarcity of suitable datasets for training AI models on raw high-resolution satellite images, the researchers developed an innovative simulation workflow. This workflow generates ‘raw-like’ products from existing high-resolution L1 imagery, enabling a systematic evaluation of how raw data impacts deep learning models. The process involves several key steps:
- Degradation: A sensor simulator tool reverses the restoration process, transforming L1 products into degraded raw products (L0) by adding noise and applying other sensor-like effects.
- Restoration: Neural networks, specifically the Enhanced Deep Super-Resolution network (EDSR), are used to restore raw images, mimicking the traditional processing chain. This step is skipped when simulating a raw image.
- Pansharpening: The Brovey method is applied to merge high-resolution panchromatic images with upsampled low-resolution multispectral images, creating high-resolution multispectral products.
- Tiling: To ensure vessels of interest are fully captured, a new tiling algorithm centers each vessel within a patch, with a random offset to prevent overfitting.
The study utilized a custom-built database of high-resolution bundle products from MAXAR, featuring over 24,000 annotated ships across 53 vessel classes. This dataset was used to create three types of images: reference L1, simulated L1 (ËœL1), and simulated raw datasets.
Evaluating AI Performance on Raw Data
For object detection and classification, the researchers selected two small, single-stage deep learning models known for their compatibility with embedded hardware: YOLOv11s and YOLOX-S. These models were trained on the different datasets to assess the validity of the simulation workflow and the impact of raw images.
Initial experiments revealed that classifying very small vessels at the study’s 2.0-meter resolution was challenging. Consequently, the focus shifted to the six largest vessel classes to ensure accurate evaluation.
Key Findings
The results provided crucial insights:
- Simulation Workflow Validation: Models trained on simulated ËœL1 data showed only a slight decrease in performance compared to those trained on reference L1 data. This indicates that the simulation workflow effectively generates data that closely represents original L1 products, making it suitable for evaluating AI models.
- Impact of Raw Products: While models trained on raw data performed similarly to those trained on ËœL1 data at low to medium confidence thresholds, their performance deteriorated more rapidly at higher confidence levels. This suggests that raw data degradation primarily affects the robustness and precision of predictions.
- Boundary Identification Challenges: Explainability maps, generated using the Xplique library, revealed that models trained on raw data struggled to accurately identify the boundaries of objects. In contrast, models trained on original products could extract vessel shapes with greater precision. This difficulty in precise boundary localization contributes to less reliable predictions, especially for complex objects or edge cases.
Also Read:
- Aligning AI Communication: Semantic Equalization in DeepJSCC
- Unifying Perception and Action: A Deep Dive into Vision-Language-Action Models for Robotics
Future Directions
The study concludes that while object detection capabilities remain largely intact with raw images, the degradation significantly impacts the precision and confidence of predictions, particularly concerning object boundaries. These findings are vital for identifying the most impactful degradations and guiding future research. The authors plan to further investigate model predictions using explainable AI tools, work on classifying smaller vessels more efficiently, and develop novel AI architectures specifically designed to mitigate the effects of raw data degradation. The ultimate goal is to efficiently deploy AI models directly onboard satellites, enhancing autonomous remote sensing capabilities for applications like maritime monitoring and environmental protection.
