TLDR: This research presents an FPGA-based implementation of real-time semantic segmentation for autonomous vehicles. It leverages the LMIINet architecture, optimized with Quantization-Aware Training (QAT) for 8-bit precision, and the CGRA4ML hardware framework. The system achieves approximately 90% pixel accuracy and 45% mIoU on the Cityscapes dataset, operating at 20 FPS with 50.1 ms latency on a ZCU104 FPGA. This approach demonstrates a significant reduction in power consumption compared to GPU solutions while maintaining competitive accuracy and real-time performance, offering a viable path for energy-efficient deep learning in autonomous driving.
Autonomous vehicles rely heavily on understanding their surroundings in real-time. One crucial aspect of this understanding is called semantic segmentation, which involves classifying every single pixel in an image to identify objects like roads, sidewalks, vehicles, and pedestrians. While highly accurate, these systems often demand significant computational power, making them challenging to implement in real-time within the strict power and cost limits of self-driving cars.
A new research paper introduces an innovative approach to tackle this challenge by implementing a lightweight neural network called LMIINet on Field-Programmable Gate Arrays (FPGAs) using a specialized hardware framework known as CGRA4ML. This work aims to achieve high accuracy in semantic segmentation while operating efficiently in real-time.
Why FPGAs for Autonomous Driving?
Traditional Graphics Processing Units (GPUs) are powerful but can be power-hungry and introduce unpredictable delays, which are problematic for safety-critical applications like autonomous driving. FPGAs offer a compelling alternative due to their ability to provide fine-grained parallel processing and precise control over memory. This allows for deterministic latency, meaning delays are predictable and consistent, unlike GPUs where operating system and driver overheads can cause jitter. FPGAs also reduce data movement by keeping computations on-chip and support efficient fixed-point arithmetic, which is crucial for power savings.
LMIINet and CGRA4ML: A Powerful Combination
The researchers adapted the LMIINet architecture, a model known for balancing accuracy and efficiency, specifically for FPGA deployment. They made several modifications to make it hardware-friendly, such as simplifying complex connections, using operations like depthwise-separable convolutions, and redesigning parts of its Flatten Transformer module. The model was trained using Quantization-Aware Training (QAT) with 8-bit precision, which significantly reduces memory usage and enables faster, more energy-efficient computations.
The Coarse-Grained Reconfigurable Array for Machine Learning (CGRA4ML) framework plays a pivotal role here. Unlike other FPGA tools that struggle with larger models, CGRA4ML allows layers to stream data off-chip, supporting more complex networks. It generates SystemVerilog code, which can be deployed flexibly on FPGAs or even hardened into Application-Specific Integrated Circuits (ASICs) for mass production, offering a clear path for future cost and energy reductions.
Real-Time Performance and Accuracy
The FPGA-based LMIINet implementation achieved impressive results on the Cityscapes dataset, a benchmark for urban scene understanding. It reached approximately 90% pixel accuracy and 45% mean Intersection-over-Union (mIoU), which are competitive with unquantized GPU models. Crucially, it operates in real-time at 20 frames per second (FPS) with a latency of 50.1 milliseconds on a Xilinx ZCU104 FPGA board. This performance meets the stringent requirements for autonomous vehicle applications.
Also Read:
- Autonomous Vehicles Gain Self-Awareness: Predicting the Road Ahead with Quantified Uncertainty
- Advanced LiDAR System Enhances Urban 3D Change Detection for Smart City Maps
Comparison with Other Solutions
When compared to GPU solutions, the FPGA implementation offers a significant reduction in power consumption. While some other FPGA designs, like ENetHQ, achieve even lower latency, they often do so at the cost of significantly lower accuracy (e.g., 36.8% mIoU for ENetHQ compared to LMIINet’s 45% mIoU). This research demonstrates a better balance between accuracy, speed, and power efficiency, making it a strong candidate for edge automotive systems where power and thermal constraints are critical.
The work highlights that FPGAs, when combined with optimized neural network architectures and frameworks like CGRA4ML, can deliver high-performance, low-latency semantic segmentation. This paves the way for more scalable and energy-efficient deep learning deployments in real-world autonomous driving scenarios. You can read the full research paper for more details. Read the full paper here.
