TLDR: This research investigates how Vision Language Models (VLMs) can be used for closed-loop symbolic planning in robotics. The study finds that closed-loop planning is superior to open-loop for geometric task completion, even in static environments. While more frequent replanning (shorter control horizon) doesn’t always yield significant benefits, providing the VLM with previous plans and execution feedback (warm-starting) is consistently crucial for improving overall performance and reducing errors. The paper recommends using warm-started closed-loop planners with an appropriately chosen control horizon.
Robotics is an exciting field, and getting robots to perform complex tasks reliably is a major challenge. A key area of research involves “symbolic planning,” where robots use high-level instructions to decide a sequence of actions. Recently, powerful AI models like Large Language Models (LLMs) and Vision Language Models (VLMs) have shown great promise in this area, especially for understanding and reasoning about the world.
However, using these advanced AI models for “closed-loop symbolic planning” – where the robot continuously updates its plan based on new information – is still largely unexplored. Because LLMs and VLMs can sometimes act like “black boxes” and produce unexpected errors, integrating them into critical robotic planning systems can be tricky and costly. This is where a new research paper, authored by Hao Wang, Sathwik Karnik, Beatrice Lim, and Somil Bansal, steps in. Their work investigates how to effectively use VLMs as closed-loop symbolic planners for robotic applications, specifically from a control-theoretic perspective.
Understanding the Core Concepts
To make sense of their findings, it’s helpful to understand a few key terms:
Open-loop Planner: Imagine a robot that gets a complete list of instructions at the very beginning and then just follows them, no matter what happens. It doesn’t adjust its plan if something unexpected occurs.
Closed-loop Planner: This is like a more adaptive robot. It gets an initial plan, but it’s ready to generate new plans (or “replan”) as it executes actions and receives updated information about its environment.
Control Horizon: This refers to how many actions a closed-loop planner executes before it pauses to consider replanning. A “shorter” control horizon means the robot replans more frequently, while a “longer” horizon means it executes more actions before checking in.
Warm-Starting: This is like giving the planner a head start. Instead of starting from scratch every time it replans, warm-starting provides the planner with its previous plan and information about how those actions were executed.
The Experiment: Testing VLMs in Action
The researchers set up a series of controlled experiments across four different robotic environments, ranging from simple cube manipulation (CUBE-EASY) to more complex tasks involving household objects and specific logical constraints (YCB-HARD). They compared two main types of planners: an open-loop planner and a closed-loop planner, which could replan after a certain number of actions or if an action failed. They tested these planners with three different Vision Language Models: GPT-4.1-mini, Gemini-2.5-flash, and Llama-4-Maverick-17B, running 50 trials for each scenario.
The performance was measured using several metrics, including the “Task Completion Rate” (how often the robot successfully finished the entire task) and the “Goal Achieved Rate” (how often objects were placed correctly, even if logical constraints weren’t fully met). They also looked at how well the planners handled logical reasoning and corrected errors.
Key Findings and Insights
The study yielded several important observations:
Closed-Loop Planning is Generally Better: Even in static environments where objects don’t move unless the robot interacts with them, closed-loop planners performed significantly better than open-loop planners in terms of successfully placing objects (Goal Achieved Rate). This is because closed-loop planners get a chance to correct geometric errors that the VLM might make during its initial planning. However, for complex logical reasoning errors, a single replanning opportunity wasn’t always enough to show a statistically significant improvement.
Shorter Control Horizon Isn’t Always the Answer: Intuitively, one might think that replanning more frequently (a shorter control horizon) would always lead to better performance. However, the experiments showed that while shorter horizons often led to the best task completion rates, these improvements were not consistently statistically significant. The researchers suggest that while more frequent replanning offers more chances for correction, it also provides more opportunities for the VLM to make new mistakes. Ultimately, the inherent reasoning capability of the VLM itself seemed to be a more critical factor for logical reasoning than the frequency of replanning.
Warm-Starting is Highly Beneficial: This was a clear winner. Providing the closed-loop planner with its previous plan and feedback on execution status (warm-starting) consistently improved overall task completion and geometric reasoning. In some cases, without warm-starting, the performance of the planners “completely collapsed,” highlighting its importance. Warm-starting helped planners make fewer negative logical corrections (i.e., making things worse). However, it also meant planners were less likely to make positive logical corrections, as they tended to stick closer to the previously generated plan, even if it had minor flaws.
Also Read:
- Multimodal Diffusion Forcing: A Unified AI Framework for Robust Robot Manipulation
- Unpacking Object-Centric World Models: A Deep Dive into DLPWM’s Strengths and Stumbling Blocks in Policy Learning
Recommendations for Future Robotic Systems
The authors conclude with a straightforward recommendation: for robotic applications, it is highly advisable to use closed-loop planners with warm-starting whenever possible. When choosing a control horizon, the focus should be on ensuring the system is reactive enough for the task, rather than simply replanning as frequently as computation allows. The specific Vision Language Model used also plays a significant role in the planner’s overall performance.
This research provides valuable insights into making VLM-powered robotic planning more robust and reliable, paving the way for more capable autonomous systems. You can read the full paper here: Using Vision Language Models as Closed-Loop Symbolic Planners for Robotic Applications: A Control-Theoretic Perspective.
