TLDR: Fast-SEnSeI is a new, lightweight, sensor-independent module designed for efficient cloud segmentation directly on satellites. It processes multispectral imagery from various sensors, producing fixed-size feature maps for a compact segmentation model deployed on FPGA. This CPU-FPGA hybrid approach overcomes limitations of ground-based processing, enabling real-time cloud detection with high accuracy and reduced resource consumption on embedded satellite hardware.
As our planet is continuously monitored by an increasing number of satellites, the sheer volume of data collected presents a significant challenge for traditional ground-based processing. Sending all raw imagery back to Earth for analysis creates bottlenecks due to limited bandwidth and the need for timely information. This has spurred the development of artificial intelligence capabilities that can operate directly on satellites, known as on-board AI.
One particularly impactful application of on-board AI is cloud segmentation. Clouds frequently obscure large portions of Earth observation imagery, rendering the data unusable or less valuable for many tasks. It’s estimated that clouds cover about 70% of the Earth’s surface at any given time. The ability to detect clouds in orbit and assess image quality in real-time allows satellites to make smarter decisions about what data to downlink, conserving bandwidth and speeding up the delivery of high-quality, clear-sky observations to users.
However, deploying AI on satellites comes with strict limitations: embedded hardware has limited power and computational resources. This necessitates highly efficient models. Another major hurdle is sensor generalization. Most existing deep learning models for remote sensing are designed for specific sensors, meaning differences in the number, placement, and width of spectral bands between sensors make models difficult to reuse across different satellite platforms. Sensor independence aims to solve this by allowing a single model to process data from various sensors with different spectral profiles.
Introducing Fast-SEnSeI: A Lightweight Solution for On-board Cloud Masking
A new research paper introduces Fast-SEnSeI, a lightweight, sensor-independent encoder module specifically designed for efficient on-board cloud segmentation. Building on previous work like SEnSeI-v2, Fast-SEnSeI integrates an improved spectral descriptor, a more compact architecture, and robust handling of missing spectral bands. It can accept any combination of spectral bands and their corresponding wavelengths, producing standardized feature maps that feed into a small, quantized segmentation model.
The Fast-SEnSeI module itself runs efficiently on embedded CPUs using Apache TVM, a machine learning compiler that optimizes models for various hardware. The segmentation model, based on a modified U-Net architecture, is then deployed on an FPGA (Field-Programmable Gate Array). This CPU-FPGA hybrid pipeline is well-suited for the demanding environment of space-qualified hardware, such as the Zynq 7000 and Ultrascale+ families.
Key Innovations and Design Choices
Fast-SEnSeI introduces several targeted changes to achieve its efficiency and accuracy:
- Improved Descriptor Design: The way spectral characteristics of each band are encoded has been refined. The new approach uses a standard Transformer formulation for wavelength normalization and encoding, making it more expressive and sensitive to variations across the wavelength range. It also uses TVM-compatible summary statistics (minimum, maximum, mean, standard deviation) for band-level reflectance instead of percentiles, enabling hardware deployment without performance loss.
- Optimized Architecture: The module combines the processing core of SEnSeI-v2 with a more efficient output block from SEnSeI-v1. This change, along with reducing the number of output feature maps, significantly boosts processing speed without compromising accuracy.
- Reduced Model Size: The spectral encoding was shrunk from 64 to 32 dimensions, and the overall network architecture was made more compact. This reduced the number of trainable parameters from 355,000 to 117,000, leading to faster processing and a smaller memory footprint on embedded systems.
- Robust Padding-Band Handling: During training, the model is exposed to random subsets of bands, with missing bands represented by ‘padding-bands’. Fast-SEnSeI ensures that these padding-bands do not negatively influence the model’s learning or inference, especially when only a few bands are available.
Lightweight Segmentation Network
The segmentation model itself is designed for resource-constrained FPGAs. It uses a modified U-Net-like architecture with a focus on balancing operations across layers for optimal inference speed and hardware utilization. To further reduce its footprint, the model is quantized using Quantization-Aware Training (QAT), where weights and activations are represented with fewer bits (e.g., 4-bit or 8-bit), leading to a compact and hardware-efficient model with minimal accuracy degradation.
Performance and Generalization
Evaluations on Sentinel-2 and Landsat 8 datasets demonstrated that Fast-SEnSeI achieves accurate cloud segmentation across diverse input configurations. It performs reliably even with a single input band, with performance steadily improving as more bands are added. Crucially, the model maintains strong performance when trained on multiple datasets and evaluated on different band combinations, a capability traditional fixed-band models lack.
Compared to its predecessor, SEnSeI-v2, Fast-SEnSeI delivers higher throughput and a smaller memory footprint without sacrificing accuracy. For instance, it can process a 512×512 image with five spectral bands in approximately 30 milliseconds on an ARM Cortex-A53 processor, making it suitable for real-time on-board use. The entire pipeline, including the FPGA-accelerated segmentation, can complete processing in under 354 milliseconds.
Also Read:
- Smart Vision: How AI is Enhancing Object Detection in Challenging Environments
- New Hyperspectral Adapter Boosts Semantic Segmentation for Autonomous Driving
Conclusion
Fast-SEnSeI represents a significant step forward in on-board Earth observation. By providing a lightweight, sensor-independent module for cloud masking, it enables satellites to perform critical preprocessing directly in orbit. This not only conserves valuable downlink bandwidth but also accelerates the delivery of high-quality data for various Earth observation tasks. The generic nature of the module means it can support any downstream model, extending its utility beyond just cloud segmentation.
For more details, you can refer to the full research paper: Fast-SEnSeI: Lightweight Sensor-Independent Cloud Masking for On-board Multispectral Sensors.
