TLDR: The paper introduces a new reinforcement learning framework called Sequence-Aware Soft Actor-Critic (SA-SAC) for optimizing engine control in series hybrid electric vehicles (HEVs). By integrating Gated Recurrent Units (GRUs) and Decision Transformers (DTs) into the SAC algorithm, the SA-SAC agents can better capture temporal driving patterns. Experiments show these sequence-aware agents significantly reduce fuel consumption and improve generalization compared to traditional feedforward networks, performing close to optimal Dynamic Programming solutions, especially in varied and unseen driving conditions.
As the world increasingly shifts towards sustainable transportation, hybrid electric vehicles (HEVs) are becoming a crucial part of the heavy-duty truck sector. These vehicles offer a promising path to reducing fuel consumption and greenhouse gas emissions. However, effectively managing the energy distribution between the engine and battery in real-time, especially under unpredictable driving conditions, remains a significant challenge.
Historically, various methods have been employed for energy management in HEVs, ranging from simple rule-based strategies to more complex approaches like Dynamic Programming (DP), Equivalent Consumption Minimization Strategy (ECMS), Model Predictive Control (MPC), and Pontryagin’s Minimum Principle (PMP). While these methods have their merits, many are computationally intensive, making them impractical for real-time applications.
Recent advancements in computational power and artificial intelligence have paved the way for data-driven energy management. Reinforcement Learning (RL) algorithms, in particular, have shown great promise. Among these, the Soft Actor-Critic (SAC) algorithm stands out. Unlike traditional RL methods that solely aim to maximize rewards, SAC also maximizes policy entropy, which encourages exploration and helps in finding near-optimal solutions in complex, continuous control environments.
However, a common limitation of many existing RL-based HEV controllers is their reliance on feedforward neural networks, which often overlook the crucial temporal dependencies inherent in driving patterns. Driving is a sequential process, and past actions and states significantly influence future decisions. To address this, a new research paper introduces an innovative framework called Sequence-Aware SAC (SA-SAC).
Enhancing SAC with Sequence Awareness
The core innovation of SA-SAC lies in integrating advanced sequence-aware network architectures into the SAC algorithm: Gated Recurrent Units (GRUs) and Decision Transformers (DTs). GRUs are a type of recurrent neural network designed to capture short- and mid-term temporal patterns by using a memory mechanism that updates hidden states over time. This allows the system to remember relevant past information when making current decisions.
Decision Transformers, on the other hand, reformulate reinforcement learning as a sequence modeling problem. They use a causal attention mechanism to focus on relevant parts of past sequences of desired returns, states, and actions. This enables them to capture long-term dependencies, which is particularly beneficial for planning over extended periods. While DTs were originally designed for offline RL, their integration into SAC allows them to be used effectively in an online learning setting.
In the SA-SAC framework, the system observes the battery’s State of Charge (SOC), distance traveled, and power required from the electric machine. Based on this, it learns to output optimal engine speed and torque. The primary goal is to minimize fuel consumption while ensuring the battery’s SOC remains within a healthy operating range (between 15% and 85%).
Also Read:
- New Framework for Designing Memory Challenges in Reinforcement Learning Environments
- DRIVE: A New Framework for Human-Like Autonomous Driving
Experimental Validation and Key Findings
The researchers conducted extensive experiments, training the SA-SAC models under diverse conditions, including varying initial battery states, drive cycle durations, power demands, and input sequence lengths. The models were trained using the EPA Highway Fuel Economy Test (HFET) cycle and then validated on unseen drive cycles like US06 Highway (representing aggressive driving) and Heavy Heavy-Duty Diesel Truck (HHDDT) cruise segments.
The ablation studies revealed that SA-SAC agents, particularly those incorporating GRU and DT networks, significantly outperformed traditional feedforward network (FFN)-based agents. They demonstrated faster convergence, better handling of long episodes, and superior robustness when faced with varying initial conditions and power demands. This highlights their ability to generalize well to different real-world scenarios.
In validation tests, the SA-SAC agents achieved fuel savings remarkably close to the Dynamic Programming (DP) baseline, which represents a global optimal solution but is too computationally expensive for real-time use. Specifically, the SAC agent with a DT-based actor and GRU-based critic performed within 1.8% of DP in fuel savings on the HFET cycle. On unseen cycles like US06 and HHDDT, the generalized sequence-aware agents consistently outperformed FFN-based agents, showcasing their adaptability.
While the sequence-aware agents require more computational resources and the Decision Transformer can sometimes exhibit noisy outputs, their superior performance in generalization and fuel efficiency justifies their potential for real-world applications. This research marks a significant step towards more adaptive and efficient energy management systems for electrified powertrains in heavy-duty vehicles.
For more detailed information, you can read the full research paper: Sequence Aware SAC Control for Engine Fuel Consumption Optimization in Electrified Powertrain.
