TLDR: The AMAD-SRL framework integrates symbolic reinforcement learning (SRL) with a Belief-Desire-Intention (BDI) architecture to enhance autonomous UAV mission planning. This hybrid approach allows drones to adapt to complex, unforeseen situations by dynamically generating plans using PDDL, while retaining the structured reasoning of BDI. Validated in a Software-in-the-Loop environment, the framework demonstrated significant efficiency gains (up to 75% reduction in travel distance) in a target acquisition scenario, proving its ability to seamlessly combine rule-based and adaptive decision-making for robust UAV operations.
Modern autonomous drone missions are becoming increasingly complex, especially in military and industrial applications. Traditional approaches, often relying on predefined rules and plans, struggle to adapt to dynamic and unforeseen situations. This limitation can render drones ineffective when faced with unexpected challenges like communication interruptions or new threats.
A recent advancement in this field is the AMAD-SRL framework, which integrates symbolic reinforcement learning (SRL) with a Belief-Desire-Intention (BDI) architecture. This innovative approach aims to combine the robust, structured reasoning of traditional BDI systems with the adaptive, flexible decision-making capabilities of SRL. The core idea is to allow drones to dynamically generate plans in real-time when faced with complex scenarios that go beyond their pre-programmed rules.
The AMAD-SRL framework is an enhanced version of the existing Autonomous Mission Agents for Drones (AMAD) cognitive multi-agent architecture. The original AMAD framework is a comprehensive system that includes various components for autonomous operations, such as a Knowledge Store (KS) for mission data, a Context Reasoner (CR) for situational awareness, and an Autonomous Task Coordinator (ATC) that uses the BDI architecture to select and execute mission tasks. While AMAD excels at adapting plans based on real-time updates to its internal world model, its rule-based nature can limit its flexibility in highly dynamic or completely new situations.
This is where Symbolic Reinforcement Learning comes into play. SRL, utilizing the Planning Domain Definition Language (PDDL), explicitly incorporates domain-specific knowledge, safety constraints, and operational limitations into the learning process. This results in decision-making that is not only safer but also more interpretable and context-aware compared to conventional RL methods.
The AMAD-SRL framework introduces a new component called the Dynamic Planner (DP) agent. This DP agent consists of a PDDL Problem Generator and an RL-based PDDL Solver. It operates within a communication loop with the Knowledge Store and the Autonomous Task Coordinator. When the ATC encounters a complex situation that cannot be resolved by its predefined BDI plans, it dynamically invokes the Dynamic Planner. The DP then generates a PDDL problem instance based on the current situation, solves it to create an optimal plan, and stores this solution back in the Knowledge Store for the ATC to execute. This allows for a seamless transition between BDI-driven and SRL-driven planning, ensuring that symbolic planning complements BDI decision-making only when necessary.
The validation of the AMAD-SRL framework was conducted in a Software-in-the-Loop (SIL) environment. This SIL setup was designed to be structurally identical to a Hardware-in-the-Loop Simulation (HILS) platform, ensuring a smooth transition to real hardware deployment. This approach allows for realistic testing of mission planning, perception, and reasoning in the early development stages, significantly reducing the time required for field deployment.
In a specific experimental scenario, a UAV followed a predefined route. Upon simultaneously detecting a target and a threat with its onboard camera, the system interpreted this as a complex situation. This triggered a PDDL-based plan called ‘SearchAndAcquisition’. The UAV then generated a search path to cover a 3 km x 3 km area, logging the positions of threats and targets. After completing the search, the Dynamic Planner generated an optimized acquisition path, designed to avoid threats and secure the target efficiently, leading to a detailed inspection at a lower altitude.
The effectiveness of the Dynamic Planner was evaluated by comparing its performance against a baseline strategy where the UAV used a standard coverage path and simple avoidance patterns. The results were impressive: the Dynamic Planner consistently generated shorter, more threat-aware paths, leading to a measurable improvement in operational efficiency. For instance, in one representative case, mission efficiency improved by approximately 75% due to the reduction in travel distance. This demonstrated the successful integration and interoperability of the newly added Dynamic Planner with the existing AMAD framework agents.
Also Read:
- Guiding Multi-Agent Planning with Graph Neural Networks
- CAMAR: Bridging the Gap in Continuous Multi-Agent Reinforcement Learning
This research establishes a robust foundation for handling complex UAV missions by combining the strengths of structured reasoning and adaptive learning. Future work will involve developing more diverse mission scenarios and conducting both Hardware-in-the-Loop Simulation (HILS) and real flight tests to further validate and refine the system under real-world conditions. You can find more details in the full research paper: Integrating Symbolic RL Planning into a BDI-based Autonomous UAV Framework.
