TLDR: NESYRO is a neuro-symbolic framework that enhances the reliability of LLM-generated robot control code in dynamic and partially observable environments. It employs a recursive process of symbolic verification for logical consistency and interactive validation using a neuro-symbolic confidence score. When uncertainty is detected, NESYRO generates ‘safe probes’ to actively explore the environment and acquire missing observations, leading to significant improvements in task success rates and safer execution in both simulations and real-world robotic tasks.
Large language models (LLMs) have opened up exciting possibilities for controlling robots, allowing them to understand natural language instructions and generate executable code to perform tasks. This approach, often called ‘code-as-policies,’ enables robots to plan and act in flexible and adaptable ways. However, a significant challenge remains: these LLM-generated policies often struggle in real-world scenarios, especially when the environment is dynamic or not fully observable. Imagine a robot trying to clean a desk, but it doesn’t know if a drawer is locked or what’s inside – this lack of information can lead to incorrect or incomplete actions, causing tasks to fail.
To address this reliability gap, researchers have introduced NESYRO, a novel neuro-symbolic framework for embodied task planning. NESYRO aims to make LLM-generated robot code more robust by integrating explicit symbolic verification and interactive validation processes directly into the code generation pipeline. This means the robot doesn’t just generate code; it actively checks if the code makes logical sense and if it’s suitable for the current environment before executing it.
The framework operates through a recursive process involving two main phases:
Neuro-Symbolic Code Verification
In this phase, an LLM first generates a high-level task specification and the initial policy code (a sequence of skills). A symbolic tool then steps in to verify if this code logically satisfies the task specification. If there are any inconsistencies or errors, the symbolic tool provides feedback to the LLM, which then refines the code iteratively until it passes verification. This ensures that the generated code is logically sound and adheres to the task’s constraints.
Also Read:
- Bridging the Gap: How Symbolic AI Enhances Transparency and Reasoning in Large Language Models
- Co-Sight: A Framework for Trustworthy and Efficient AI Agent Reasoning
Neuro-Symbolic Code Validation
Once the code is logically verified, NESYRO moves to validate each individual skill in the policy sequence. This is where the framework assesses the environmental feasibility of the code. It uses a ‘Neuro-Symbolic Confidence Score’ (NeSyConf) for each skill. This score combines two elements: ‘Common Sense Confidence’ (CSC) from an LLM, which estimates the likelihood of a skill succeeding based on general knowledge and demonstrations, and ‘Logic Confidence’ (LC) from a symbolic validation tool, which checks if the skill’s preconditions are met in the current environment.
If a skill’s NeSyConf falls below a certain threshold, indicating uncertainty or missing information, NESYRO initiates a ‘safe probe.’ This involves generating exploratory code that actively interacts with the environment to acquire the necessary missing observations. For example, if the robot needs to open a drawer but doesn’t know if it’s locked, a safe probe might involve a gentle attempt to check the lock status. This probe code itself goes through the same verification and validation process to ensure it’s safe and grounded. Once new observations are gathered, the main policy code is refined, and the validation process continues recursively until all skills are confidently grounded.
The effectiveness of NESYRO was rigorously evaluated in both RLBench simulations and real-world settings across various task categories, including object relocation, object interaction, auxiliary manipulation, and long-horizon tasks. These experiments were conducted under different levels of ‘observability incompleteness,’ simulating environments where critical information might be sparse or ambiguous.
The results were compelling. NESYRO significantly improved task success rates by 46.2% over existing ‘Code as Policies’ baselines and achieved over 86.8% executability of task-relevant actions in real-world scenarios. It consistently outperformed other methods, especially as environmental uncertainty increased, by accurately detecting missing observations and performing safe, exploratory actions. The framework also drastically reduced ‘irreversible actions’ – mistakes that could lead to task failure or damage – from 29 to just 7 in real-world tests.
This research demonstrates a crucial step towards building more reliable and adaptable embodied agents. By combining the powerful code generation capabilities of LLMs with the robustness of symbolic reasoning and active environmental exploration, NESYRO enables robots to navigate and complete complex tasks safely and effectively, even in dynamic and partially observable real-world conditions. You can read the full research paper here: Towards Reliable Code-as-Policies: A Neuro-Symbolic Framework for Embodied Task Planning.
