TLDR: A new Deep Reinforcement Learning (DRL) model, Deep Hierarchical Cycle Planner (DHCP), has been developed to optimize traffic signal control. It uses a two-tiered approach: a high-level AI agent allocates total green light time between major directions (North-South and East-West), and low-level agents then further divide that time between straight and left-turn movements within each direction. This hierarchical system ensures predictable signal cycles while adaptively responding to traffic, leading to significantly reduced travel times and improved traffic flow compared to existing methods.
Traffic congestion is a persistent challenge in urban areas, leading to longer travel times, increased fuel consumption, and environmental pollution. Traditional traffic signal control methods often struggle to adapt to the dynamic and ever-changing flow of vehicles. In response, researchers have turned to advanced artificial intelligence techniques, particularly Deep Reinforcement Learning (DRL), to create more intelligent and adaptive solutions.
DRL allows traffic controllers to learn optimal strategies by interacting with the traffic environment in real-time. However, existing DRL-based traffic signal control methods often fall into two main categories: “choose phase” and “switch” strategies. While the “choose phase” approach offers adaptive phase selection, it can lead to unpredictable signal sequences, potentially confusing drivers and compromising safety. The “switch” paradigm, on the other hand, maintains a more predictable order but can result in unfair and inefficient allocation of green light times, favoring some movements while neglecting others.
To address these limitations, a new DRL model called Deep Hierarchical Cycle Planner (DHCP) has been proposed. This innovative framework allocates traffic signal cycle durations in a hierarchical manner, aiming to provide both predictability and efficiency. The core idea is to break down the complex decision-making process into two levels of control.
How DHCP Works: A Two-Tiered Approach
The DHCP model employs two types of agents: a high-level agent and a low-level agent, both utilizing the Deep Deterministic Policy Gradient (DDPG) algorithm, which is well-suited for making continuous decisions like allocating time durations.
The **high-level agent** acts as a strategic planner. It observes the overall traffic conditions at an intersection, specifically looking at the total number of vehicles (wave) and the length of queues on all incoming lanes. Based on this comprehensive view, it determines how to split the total signal cycle time between the two major traffic directions: North-South (NS) and East-West (EW). Its primary goal is to reduce overall traffic congestion at the intersection.
Once the high-level agent has allocated time to the NS and EW directions, the **low-level agents** take over. There’s a separate low-level agent for each major direction (NS and EW). Each low-level agent focuses on its specific direction, observing the traffic state (wave and queue length) for the straight and left-turn movements within that direction, along with the duration allocated by the high-level agent. Its task is to further divide this allocated time between the straight and left-turn movements. The objective of each low-level agent is to minimize congestion within its assigned direction.
This hierarchical structure ensures that the traffic signal planning is both globally optimized and locally refined. The high-level agent handles the broader allocation, while the low-level agents fine-tune the timing for specific movements, all while maintaining a predictable phase sequence and fixed cycle lengths, which aligns better with real-world traffic engineering practices.
Testing and Results
The DHCP model was rigorously tested on the Cityflow traffic simulator using a variety of scenarios, including real-world road networks from Jinan and Hangzhou, as well as a synthetic network. The traffic flow data included both real and artificially generated patterns, representing diverse traffic conditions.
The results were highly promising. DHCP consistently achieved superior performance compared to several baseline methods, including traditional fixed-time control, adaptive approaches like SOLT and MaxPressure, and other DRL models such as DQN, Dueling-DQN, A2C, and CoLight. While DHCP showed a slight dip in performance during the very early stages of training due to the initial adaptation required between the two hierarchical agents, it quickly converged and demonstrated the shortest average travel times across all test scenarios. This indicates its effectiveness in reducing delays and improving overall traffic flow.
Also Read:
- AI System Optimizes Vehicle Speed Classification for Urban Traffic Management
- Intelligent Vessel Routing: How AI Navigates Dynamic Waterways with Real-World Data
Conclusion and Future Directions
The Deep Hierarchical Cycle Planner (DHCP) represents a significant step forward in intelligent traffic signal control. By combining the strengths of hierarchical reinforcement learning with the DDPG algorithm, it offers a robust solution that balances predictability with adaptive efficiency. The model’s ability to allocate signal durations in a structured, two-tiered manner leads to smoother traffic flow and reduced congestion.
While the current model assumes a fixed total cycle duration and focuses on four-way intersections, future work will explore coordinating traffic signal cycles across multiple intersections and extending the model to more complex networks with diverse intersection types. For more details, you can read the full research paper here.
