TLDR: This research introduces a novel single-agent reinforcement learning framework for regional traffic signal control. It uses an adjacency matrix to represent network topology and real-time queue states, and leverages the DreamerV3 world model to learn control policies. Simulation experiments demonstrate its robust ability to reduce queue lengths and resist demand fluctuations, offering a new intelligent traffic control paradigm compatible with probe vehicle technology.
Traffic congestion is a persistent challenge in urban areas, impacting daily life, safety, the environment, and economic efficiency. A major contributor to this problem is the queuing of vehicles at intersections. While Traffic Signal Control (TSC) systems offer a promising solution, traditional optimization models often struggle to adapt to the complex and dynamic nature of real-world traffic conditions.
A recent study introduces a novel approach to tackle this issue: a single-agent reinforcement learning (RL) framework designed for regional adaptive traffic signal control. This framework aims to simplify the control process by using a centralized decision-making system, thereby avoiding the intricate coordination problems often found in multi-agent systems.
The core of this innovative model lies in its ability to unify various critical pieces of information. It uses an adjacency matrix to encode the road network’s layout, real-time queue states (derived from probe vehicle data), and current signal timing parameters. This comprehensive representation allows the system to understand the traffic environment holistically.
Leveraging the powerful learning capabilities of the DreamerV3 world model, the agent learns control policies that sequentially select intersections and adjust their signal phase splits. This process effectively regulates the flow of traffic into and out of intersections, much like a sophisticated feedback control system. The reward system is specifically designed to prioritize the dissipation of queues, directly linking congestion levels (measured by queue length) to the control actions taken by the agent.
To validate its effectiveness, the model was put to the test in simulation experiments using SUMO, a widely used traffic simulator. These experiments focused on scenarios where Origin-Destination (OD) demand fluctuated at multiple levels (10%, 20%, and 30%). The results were highly encouraging: the framework demonstrated robust resistance to these demand fluctuations and significantly reduced queue lengths across the simulated network.
This research marks a significant step forward, establishing a new paradigm for intelligent traffic control that is compatible with modern probe vehicle technology. Probe vehicles, such as ride-hailing cars and navigation system users, provide valuable real-time trajectory data that can be used to estimate crucial metrics like queue length, which are often difficult to capture with traditional sensors.
How the System Works
The methodology involves defining five key components for the RL-based TSC: state representation, action design, reward formulation, environment model, and the policy learning algorithm. The state combines congestion information (queue length) and the signal phase scheme into a single adjacency matrix. Actions involve selecting an intersection and then adjusting its signal phase split by a predefined step size, either increasing, decreasing, or keeping it the same. This sequential action design helps manage the complexity of the action space.
The reward function is based on queue length, penalizing light and heavy congestion differently to encourage both congestion mitigation and minimization of total travel time. The environment, built using Gymnasium, simulates traffic dynamics, allowing the RL agent to learn optimal strategies. DreamerV3 was chosen as the policy learning algorithm due to its data-efficient learning capabilities and its ability to predict future consequences through “imagination,” reducing the need for extensive real-world interaction.
Also Read:
- A Centralized AI Approach to Adaptive Traffic Control
- Modeling Realistic Pedestrian-Driver Interactions with Human-Like Constraints
Future Directions
Future research aims to further enhance the practical applicability of this framework. This includes incorporating stochastic OD demand fluctuations during the training phase to better simulate real-world traffic uncertainties and exploring regional optimization mechanisms for contingency events like accidents or extreme weather. These advancements will pave the way for more robust decision support in intelligent traffic management systems within complex urban environments.
For more in-depth information, you can read the full research paper here.
