TLDR: A comprehensive study evaluates how well Large Language Models (LLMs) and Vision Language Models (VLMs) provide planning feedback in various grounded environments. It finds that larger, reasoning models offer more accurate feedback, especially with advanced inference methods, and that VLMs excel in visual continuous control tasks. However, feedback quality can decline in highly complex or continuous environments, highlighting areas for future research.
A new research paper explores the effectiveness of large language models (LLMs) and vision language models (VLMs) in providing crucial feedback for planning in various simulated environments. Traditionally, training agents to follow instructions in these environments demands either meticulously designed reward functions or extensive, high-quality human demonstrations. This study investigates how foundation models, which possess vast background knowledge from their pretraining, can simplify this process by offering intelligent feedback.
The paper, titled “How well can LLMs provide planning feedback in grounded environments?”, was authored by Yuxuan Li and Victor Zhong from the David R. Cheriton School of Computer Science at the University of Waterloo. Their work provides a comprehensive analysis of how these advanced AI models perform across different types of feedback, diverse environments, and various inference methods.
The researchers evaluated five prominent types of feedback: binary feedback (whether an action is optimal), preference feedback (which of two actions is better), action advising (suggesting the correct action), goal advising (identifying the next subgoal), and delta action feedback (providing action corrections). These feedback types were tested across a range of environments, including symbolic games like CliffWalking and MiniGrid, text-based simulators such as ALFWorld and HierarchyCraft, and continuous control tasks involving a robotic arm in Robomimic.
Key findings from the study reveal that foundation models are capable of delivering high-quality feedback across these diverse domains. Notably, larger models and those with advanced reasoning capabilities consistently provided more accurate feedback. These sophisticated models also exhibited less bias and benefited more significantly from enhanced inference techniques like in-context learning and chain-of-thought prompting, which involve providing examples or guiding the model through intermediate reasoning steps.
The study also highlighted that certain types of feedback, specifically binary and preference feedback, were generally easier for the models to provide accurately, especially for less complex models. However, the quality of feedback tended to degrade in environments characterized by complex dynamics or continuous state and action spaces, which are inherently more challenging for AI to interpret and navigate.
In predominantly visual environments, such as the robotic arm control task in Robomimic, Vision Language Models (VLMs) demonstrated a clear advantage over LLMs. This is because visual observations are often more intuitive for VLMs to process than textual descriptions of complex mechanical states and actions, leading to higher accuracy in tasks like preference and binary feedback.
The researchers also delved into common types of errors made by these feedback models. Smaller models frequently struggled with formatting their responses correctly or inventing invalid answers. Errors related to insufficient knowledge were common in complex domains like HierarchyCraft, where models might make incorrect assumptions about the environment. Incorrect reasoning errors occurred when models understood the rules but failed to apply them correctly, often seen in continuous state spaces like Robomimic. Finally, incorrect visual understanding was a specific challenge for VLMs, leading to misinterpretations of images.
Also Read:
- Large Language Models Learn to Cooperate in UNO
- Small Language Models Show Promise in Formal Logic Reasoning for Ontology Engineering
This research underscores the potential of language model feedback as a promising avenue for generating learning signals in planning, potentially reducing the need for extensive hand-engineered reward functions or human demonstrations. The authors advocate for further research into eliciting robust feedback and efficiently learning from it, particularly in complex and continuous environments where traditional methods are most challenging. You can read the full paper here.
