TLDR: This research introduces a Deep Reinforcement Learning (DDPG) approach to optimize signal precoding and phase shifts in multi-Reconfigurable Intelligent Surface (RIS)-aided wireless systems. It addresses the practical challenge of amplitude-phase coupling in RIS elements and demonstrates that DDPG outperforms other methods, especially in multi-RIS setups and when trained with varying numbers of users, leading to improved spectrum efficiency for future 5G and 6G networks.
As we move towards advanced wireless communication systems like 5G and 6G, the use of millimeter-wave (mmWave) frequency bands becomes increasingly important. However, these high-frequency transmissions face significant challenges, including high power consumption and signal attenuation caused by environmental obstacles. To overcome these hurdles, Reconfigurable Intelligent Surfaces (RISs) have emerged as a promising technology. RISs are designed to reflect transmitted signals from a base station, bypassing obstacles and enhancing both spectral and energy efficiency.
A key challenge in deploying RISs lies in optimizing their performance. An RIS consists of numerous reflective elements, each capable of altering the phase of a signal. Traditionally, many studies have assumed ideal reflectivity, where the amplitude of the reflected signal remains constant. However, in practical scenarios, there’s a complex relationship where the reflecting amplitude is coupled with the phase shift, making the optimization problem much more intricate.
This research paper, titled “Deep Reinforcement Learning-Based Precoding for Multi-RIS-Aided Multiuser Downlink Systems with Practical Phase Shift”, delves into this challenge. The authors, Po-Heng Chou, Bo-Ren Zheng, Wan-Jen Huang, Walid Saad, Yu Tsao, and Ronald Y. Chang, propose a novel approach to jointly optimize the transmitter precoding and the RIS phase shift matrix to maximize spectrum efficiency in multi-RIS-aided multiuser downlink systems, specifically considering this practical coupling effect.
The Optimization Challenge
The problem of designing and optimizing precoding and RIS phase shift matrices in a multi-RIS-aided multiuser multiple-input single-output (MISO) system is inherently non-convex. Traditional optimization techniques, such as alternating optimization (AO), often struggle with high computational complexity, which increases significantly with the number of optimization parameters (like antennas, RISs, and reflective elements). This leads to slow, outdated solutions, especially in dynamic wireless mmWave channels.
Deep Reinforcement Learning to the Rescue
To address these limitations, the researchers propose using Deep Deterministic Policy Gradient (DDPG), a type of Deep Reinforcement Learning (DRL). DDPG is particularly well-suited for dynamic environments because it can adapt by learning from experience. Unlike some other DRL methods like Deep Q Networks (DQN), DDPG can operate over continuous action spaces, which is crucial for optimizing continuous parameters like phase shifts. This allows it to approximate a full understanding of the state space more effectively.
The DDPG framework involves a policy network that explores the environment and a target network that estimates future rewards. The system learns by observing states (which include user information, previous actions, and achievable rates), performing actions (adjusting precoding and phase shifts), and receiving rewards (the sum rate of all users). This iterative learning process allows the system to find optimal strategies over time.
Also Read:
- Enhancing 6G Network Management with Unbiased AI Agent Memory
- Digital Twins Enhance 6G Communication for Mobile Augmented Reality
Key Findings and Performance
The simulation results demonstrate the effectiveness of the proposed DDPG approach. The DDPG model was trained under various scenarios, including both fixed and random numbers of users. A significant finding was that DDPG, despite its comparable complexity, consistently outperformed two optimization-based algorithms, Block Coordinate Descent (BCD) and Weighted Minimum Mean Square Error with Power Iteration (WMMSE-PI), as well as another DRL algorithm, Double DQN (DDQN).
Specifically, the study showed that:
- Multi-RIS systems significantly outperform single-RIS systems, offering 117% and 114% higher achievable rates for ideal and practical RISs, respectively, due to enhanced spatial diversity and better interference suppression.
- DDPG achieved a 63% higher achievable rate compared to DDQN in multi-RIS systems under ideal phase shift assumptions, highlighting the advantage of continuous action spaces.
- While BCD could achieve a higher rate than DDPG in some single-RIS scenarios, its computational complexity was substantially higher (7.03 × 10^8 FLOPs compared to DDPG’s 3.71 × 10^6 FLOPs for online inference), making DDPG a more practical solution for real-time applications.
- Training the DDPG model with a random number of users yielded superior performance compared to training with a fixed number of users. This indicates that the model trained with diverse user scenarios is more robust and adaptable to varying network conditions.
In conclusion, this research presents a robust DDPG-based framework for optimizing multi-RIS-aided downlink systems, taking into account the crucial practical aspect of phase shift limitations and amplitude-phase coupling. The proposed method demonstrates strong performance and adaptability, paving the way for more efficient and reliable wireless communication in future 5G and 6G networks. Future work aims to extend this algorithm to more advanced systems like Simultaneously Transmitting and Reflecting (STAR)-RIS-aided MIMO systems with user mobility considerations.
