TLDR: A new research paper introduces Multimodal Policy Internalization (MPI), a method to embed complex, reasoning-intensive multimodal policies directly into AI models. This allows conversational agents to follow instructions more accurately and efficiently without needing the policy in every prompt. The paper presents two new datasets (ClevrPolicy and GTAPolicy) and a three-stage training framework called TriMPI, which includes visually-masked continual pretraining, supervised fine-tuning, and reinforcement learning with PolicyRollout. TriMPI shows significant improvements in policy following, generalization, and efficiency, reducing prompt tokens and inference time.
Conversational AI agents like ChatGPT and Alexa+ have become a staple in our daily lives. These advanced systems often rely on predefined “policies” – essentially, a set of instructions that dictate how the AI should behave, what style of response to use, and how to utilize various tools. Traditionally, these policies are fed to the AI as part of the input prompt, known as “in-context prompts.”
However, as these policies grow in complexity and length, they present significant challenges. They can become so long (ranging from 1,000 to 50,000 tokens) that the AI struggles to follow them accurately. This also incurs a substantial fixed computational cost for every interaction, regardless of how short the user’s actual query is. For instance, a policy might be 20 to 250 times longer than a typical user query, leading to high processing costs.
The emergence of multimodal conversational agents, which can process and respond to information from various sources like text and images, further complicates this. Policies for these agents need to handle multimodal tasks and even visual instructions, an area that has received little attention in previous research. Existing prompt compression techniques primarily focus on simpler task templates, not the complex reasoning required for policies. Similarly, prior work on policy alignment has been limited to text-only safety instructions.
Introducing Multimodal Policy Internalization (MPI)
To address these critical gaps, researchers have introduced a new task called Multimodal Policy Internalization (MPI). The core idea behind MPI is to embed these reasoning-intensive, multimodal policies directly into the AI model’s parameters. This means the model learns and “internalizes” the policy, allowing it to follow instructions more effectively without needing the policy to be explicitly included in every input prompt during inference. This approach promises stronger policy-following behavior and improved efficiency.
MPI comes with its own set of unique challenges. First, it targets policies that demand complex reasoning across multimodal tasks, such as decision-making and tool usage. Second, there’s a significant lack of existing datasets that feature multimodal question-answer pairs adhering to predefined policies. Third, established training methods for internalizing multimodal policies are scarce.
New Datasets for Multimodal Policies
To overcome the data challenge, the researchers constructed two novel datasets: ClevrPolicy and GTAPolicy. ClevrPolicy is designed for internalizing complex decision-making policies that require multi-step reasoning. It’s built upon the synthetic CLEVR dataset, offering flexible control over policy complexity and dataset size, which is ideal for evaluating different algorithms. GTAPolicy, on the other hand, focuses on multimodal tool-usage instructions using real-world images and user queries, specifically addressing scenarios with limited available data.
Also Read:
- Unpacking the Architecture of Autonomous LLM Agents
- New Memory System Enables Smarter, More Adaptable GUI Agents
TriMPI: A Three-Stage Training Framework
A key innovation presented in this work is TriMPI, a novel three-stage training framework designed to internalize multimodal policies effectively. The challenge lies in aligning the AI’s responses with a policy that isn’t present during inference. TriMPI tackles this by:
- Visually-Masked Continual Pretraining (VM-CPT): This initial stage directly injects policy knowledge into the model’s parameters by training it on the multimodal policy context. It helps the model “warm up” to the policy before fine-tuning.
- Supervised Finetuning with Chain-of-Thought (CoT SFT): In this stage, the model is fine-tuned on data where it explicitly reasons through the policy rules before generating an answer.
- Reinforcement Learning (RL) with PolicyRollout (PoRo): This crucial stage uses reinforcement learning to enable the model to learn from a wider range of policy-related responses through trial and error, rather than just memorization. PolicyRollout is a simple yet effective extension that augments the learning process by including policy-aware responses during the “rollout” phase, ensuring more grounded exploration without creating a mismatch between training and inference.
The results show that TriMPI significantly improves MPI performance, achieving substantial gains over traditional supervised fine-tuning and even in-context policy following. It also demonstrates enhanced generalization to policy updates and robustness against catastrophic forgetting, meaning the model retains its general abilities while learning specific policies. This approach leads to impressive reductions in prompt tokens (up to 93.9%) and prefill inference time (up to 85.7%), making conversational agents more efficient.
This research marks a significant step as the first work on multimodal policy internalization, providing essential datasets, training recipes, and comprehensive evaluations. The code and data will be made publicly available for further research, laying a strong foundation for future advancements in making conversational agents smarter and more efficient. You can find the full research paper here.
