TLDR: This paper introduces a novel framework for training Deep Reinforcement Learning (DRL) models to optimize resource allocation in distributed IoT systems. Unlike previous research, this framework focuses on practical implementation, using real-world feedback (Acknowledgment information) from actual data transmissions to train DRL models. The system, involving IoT end devices, a gateway, and a personal computer for training, demonstrates improved communication efficiency, achieving a 91% Frame Success Rate in testing, a 17% improvement over traditional methods.
The rapid expansion of Internet of Things (IoT) devices has brought about incredible opportunities across various sectors, from smart cities to healthcare. However, this growth also introduces significant challenges, particularly spectrum congestion, which can hinder efficient communication. Ensuring high-quality communication for a vast number of IoT devices requires effective resource allocation methods.
While centralized resource allocation relies on a single processor, distributed approaches offer enhanced spectrum and energy efficiency by allowing devices to make independent decisions. Deep Reinforcement Learning (DRL) has emerged as a powerful tool for tackling complex decision-making problems in distributed resource allocation, but its practical implementation, especially with real-world data, has remained a significant hurdle.
A new research paper, titled “Deep Reinforcement Learning Based Resource Allocation in Distributed IoT Systems” by Aohan Li and Miyu Tsuzuki, addresses this gap by proposing a novel framework for training DRL models in actual distributed IoT environments. This framework focuses on using real-world feedback, specifically Acknowledgment (ACK) information from data transmissions, to train DRL models for channel selection.
How the Framework Works
The proposed framework operates in two main phases: training and testing. In the training phase, IoT end devices (EDs) collaborate with a Personal Computer (PC) to train their DRL models. The PC executes the DRL model for each ED, performing channel selection before each data transmission. The selected channel is then communicated to the ED, which uses it to transmit data to a Gateway (GW).
Crucially, after transmission, the ED receives feedback: either an ACK if the transmission is successful, or information about a failure. This feedback is then sent back to the PC to update the DRL model. This iterative process continues for a set number of training iterations, allowing the DRL model to learn and adapt based on real-world outcomes. The DRL components are defined with the IoT ED as the agent, the state and reward determined by ACK information (1 for success, 0 for failure), and channel selection as the action. Double Deep Q-Learning is employed to enhance learning stability and performance.
Once the training phase is complete, the testing phase begins. In this phase, the now well-trained DRL model on each ED is directly used for channel selection during data transmission, without the need for the PC’s assistance in real-time decision-making.
Also Read:
- Bridging the Gap: Cloud-Edge-Terminal Intelligence for the AI-Powered Internet of Things
- Spiking Neural Networks Power Energy-Efficient Beamforming for Next-Gen V2X Communication
Practical Implementation and Results
The researchers implemented this framework using two IoT EDs and a GW. The GW was configured to receive data on two specific frequencies (922.8 MHz and 923.2 MHz), while the EDs could select from three available channels (922.4 MHz, 922.8 MHz, and 923.2 MHz). Both the IoT EDs and the GW were equipped with Arduino Nano controllers and Long Range (LoRa) modules, specifically ES920LR, demonstrating a practical, low-cost setup.
The performance evaluation, measured by the Frame Success Rate (FSR), showed promising results. During the training phase, the average FSR of the IoT EDs gradually increased, confirming that the DRL method effectively learns from environmental feedback. In the testing phase, the trained model achieved an impressive average FSR of 91% over 100 episodes. This represents a significant 17% improvement compared to the traditional ϵ-greedy method, highlighting the effectiveness of the DRL-based approach in optimizing resource allocation.
This research successfully demonstrates the feasibility and effectiveness of implementing DRL-based resource allocation in distributed IoT systems using real-world data. Future work aims to further accelerate the convergence of the training phase, enhance system stability, and extend this approach to high-density, scalable, and distributed IoT networks. For more details, you can refer to the full paper here.
