TLDR: ReTiDe is a hardware-accelerated denoising system that uses quantized deep learning models on FPGAs to provide real-time, energy-efficient video denoising. It achieves significantly higher throughput and energy efficiency than previous FPGA solutions, with negligible quality loss, and integrates seamlessly into professional workflows like NUKE. The system offloads computationally intensive tasks to networked FPGA services, making advanced denoising more practical and scalable for high-resolution media processing.
Denoising is a fundamental process in modern video production, essential for everything from improving video codec performance by suppressing sensor noise and compression artifacts to cleaning up footage in cinematic post-production. However, the most advanced deep learning denoisers are incredibly demanding computationally. They typically rely on powerful GPUs, which can be expensive and consume a lot of power, especially when dealing with real-time, high-resolution video streams.
Addressing this challenge, researchers have introduced Real-Time Denoise, or ReTiDe, a new hardware-accelerated denoising system designed for energy-efficient motion picture processing. ReTiDe leverages Field Programmable Gate Arrays (FPGAs) in data centers to handle the intensive computations.
How ReTiDe Works
The core of ReTiDe is a compact convolutional neural network model. To make it highly efficient for hardware, this model undergoes a process called quantization. Specifically, it’s converted from a high-precision FP32 format to a lower-precision INT8 format using Post-Training Quantisation (PTQ) and then fine-tuned with Quantisation-Aware Training (QAT). This allows the model to be compiled and run efficiently on AMD Deep Learning Processor Unit (DPU)-based FPGAs.
ReTiDe is designed with a client-server architecture. This means that computationally heavy denoising tasks can be offloaded from a user’s local CPU or GPU to a networked FPGA service. A significant advantage of this approach is its seamless integration with existing professional workflows, such as those using NUKE software, without disrupting the artist’s tools or creative process.
Performance and Efficiency
The performance of ReTiDe is impressive. On standard benchmarks, it achieves a throughput of 37.71 Giga Operations Per Second (GOPS), which is significantly higher than previous FPGA denoising accelerators. More importantly, it boasts 5.29 times higher energy efficiency, meaning it uses much less power to achieve its results. Crucially, these gains in speed and efficiency come with negligible degradation in image quality, as measured by metrics like Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index (SSIM).
This indicates that specialized hardware accelerators like ReTiDe can provide a practical and scalable solution for denoising in both video encoding pipelines and post-production. It allows for a substantial reduction in energy consumption per frame without compromising visual quality or compatibility with current production workflows. The code for ReTiDe is also made available as an open-source project.
Beyond Traditional Denoising
The paper also delves into the evolution of image denoising. Historically, methods like BM3D and wavelet-based filters were common. More recently, deep learning approaches, including various CNN architectures like DnCNN and U-Net, have shown superior performance in restoring image details. The latest advancements involve transformer networks, which can capture long-range dependencies in images more effectively.
While GPUs have been the go-to for accelerating these deep learning models, FPGAs offer unique advantages in terms of latency, power efficiency, and reconfigurability. Previous FPGA efforts often focused on simpler, classical filters or less flexible deep learning models. ReTiDe distinguishes itself by implementing a blind denoiser (meaning it doesn’t need to know the type or intensity of noise beforehand) for both color and grayscale images, leveraging highly parallel DPU acceleration.
Also Read:
- Decoding Actions: A Smarter Way to Analyze Compressed Videos
- Optimizing Deep Neural Network Quantization for Efficiency
Integration and Future Outlook
The integration with professional software like NUKE is a key feature. A custom plugin allows NUKE to send large data chunks to the FPGA server, where a pre-processor segments and batches the media for parallel processing across multiple DPUs. The Vitis AI toolchain facilitates this entire process, from model quantization to deployment on AMD Alveo U50 FPGA cards.
ReTiDe demonstrates that it can achieve denoising quality comparable to advanced FP32 models for both color and grayscale images. Its significant improvements in energy efficiency and throughput over existing FPGA-based deep learning denoisers pave the way for accelerating and offloading complex image processing algorithms in professional settings. Future work aims to further enhance detail preservation using mixed-precision quantization and explore model sparsification for even greater energy efficiency. You can find more details about this research in the full paper available here.
