TLDR: PeBR-R1 is a new vision-language model (VLM) that uses a two-stage reinforcement learning framework to improve both visual perception and reasoning. Unlike previous methods that directly applied techniques from language models, PeBR-R1 first enhances the model’s ability to understand visual inputs (Perception RL) using detailed image descriptions and keyword matching, then focuses on improving its logical reasoning (Reasoning RL) with accuracy and format rewards. This sequential training, combined with smart data sampling, helps the model overcome challenges like ‘vanishing advantage’ in RL. PeBR-R1 has shown superior performance on various visual reasoning benchmarks, outperforming many existing open-source and even some closed-source VLMs.
Vision-language models (VLMs) are designed to understand and reason about both images and text. While reinforcement learning (RL) has significantly boosted the reasoning abilities of large language models (LLMs), directly applying these techniques to VLMs has proven challenging. The core issue is that VLMs must first accurately ‘see’ and interpret visual information before they can effectively reason about it. Many existing VLM approaches tend to overemphasize language cues, sometimes neglecting crucial visual details.
To tackle this, researchers from Tsinghua University and Baidu Inc. have introduced a novel framework called PeBR-R1, which stands for Perception Before Reasoning. This innovative approach uses a two-stage reinforcement learning process specifically designed to enhance both the visual perception and reasoning capabilities of VLMs. The paper, titled Perception Before Reasoning: Two-Stage Reinforcement Learning for Visual Reasoning in Vision-Language Models, details how this method leads to significant performance improvements.
A Two-Stage Approach to Learning
The PeBR-R1 framework is built on the idea that perception and reasoning should be optimized sequentially rather than simultaneously. This prevents interference between the two complex learning objectives.
The process begins with a ‘warm-up’ phase, where the base VLM is fine-tuned using a large dataset to give it initial visual understanding and reasoning skills. To ensure robust learning, the training data is carefully sampled based on how well the model performs on different questions. Questions are categorized into ‘Easy,’ ‘Medium,’ and ‘Hard’ cases depending on the number of correct responses generated by the model.
Stage 1: Sharpening Visual Perception
The first stage of reinforcement learning focuses entirely on improving the model’s visual perception. For this, the framework uses ‘Easy cases’ – questions where the model already provides mostly correct answers. This ensures that the model is guided by reliable visual signals, minimizing the risk of learning from incorrect interpretations.
During this stage, the model is rewarded for two key aspects of visual understanding:
- Coarse-grained alignment: A ‘CLIP score’ reward measures how well the model’s generated image descriptions align with the actual input image. This uses a specialized model called FG-CLIP, which is good at capturing overall semantic correspondence.
- Fine-grained understanding: A ‘keyword reward’ encourages the model to recognize specific visual concepts like objects, numbers, attributes, and spatial relationships. This is achieved by comparing keywords extracted from the model’s description with a curated set of reference keywords.
Additionally, a ‘length penalty’ is applied to prevent the model from generating overly verbose or redundant image descriptions, ensuring concise and relevant outputs.
Stage 2: Elevating Reasoning Abilities
Once the model’s visual perception is significantly enhanced, the second stage shifts its focus to improving reasoning and problem-solving. This stage utilizes ‘Medium cases’ – questions where the model is partially correct – to provide stable learning signals for gradient-based optimization.
Here, the reward signals are rule-based and designed to promote logical consistency and accuracy:
- Format correctness: Rewards are given for following a structured, chain-of-thought response format, which helps guide the model through logical steps.
- Accuracy: The model is rewarded for providing correct final answers, directly optimizing its problem-solving performance.
Both stages employ Group-based Relative Policy Optimization (GRPO), an advanced RL algorithm, to ensure stable and effective policy updates.
Exceptional Performance Across Benchmarks
The results of PeBR-R1 are impressive. Evaluated across seven diverse multimodal reasoning benchmarks, including MathVista, MathVision, and ChartQA, PeBR-R1 consistently demonstrates superior performance. The 7B parameter version of PeBR-R1 even surpasses larger open-source models, such as Qwen2.5-VL-72B and InternVL2.5-78B, and in some cases, outperforms powerful closed-source models like GPT-4o and Claude-3.5 Sonnet.
Ablation studies confirm that the two-stage approach is crucial. Separating perception and reasoning training avoids ambiguity and allows for precise skill acquisition, leading to better outcomes than single-stage or reasoning-only methods.
Also Read:
- Test-Time Warmup: Enhancing Multimodal AI’s Visual Reasoning Capabilities
- Advancing Robot Generalization Through Preserved Vision-Language Representations
Conclusion
PeBR-R1 represents a significant step forward in enhancing the capabilities of vision-language models. By explicitly addressing the need for robust visual perception before complex reasoning, this two-stage reinforcement learning framework provides a more effective and stable training methodology. The model’s ability to accurately perceive and then logically reason about visual information bridges a critical gap, paving the way for more intelligent and reliable multimodal AI systems.
