TLDR: The research introduces REASONMAP-PLUS, a dataset providing dense rewards for fine-grained visual understanding, and REWARDMAP, a multi-stage reinforcement learning framework. REWARDMAP features a difficulty-aware reward design (including detail rewards) and a curriculum-based multi-stage RL scheme. This approach effectively tackles sparse rewards and unstable optimization in fine-grained visual reasoning for MLLMs, leading to significant performance improvements on transit map tasks and broader visual benchmarks.
Multimodal Large Language Models (MLLMs) are becoming increasingly powerful, but they still face significant hurdles when it comes to fine-grained visual reasoning, especially in complex scenarios like understanding transit maps. A recent study highlighted this challenge, showing that even advanced MLLMs struggle with spatial reasoning in structured, information-rich visual environments. This difficulty is compounded by a common problem in reinforcement learning (RL): sparse rewards, where models only receive feedback at the very end of a long reasoning process, making learning inefficient and unstable.
To tackle these issues, researchers have introduced two key innovations: REASONMAP-PLUS and REWARDMAP. REASONMAP-PLUS is an expanded dataset designed to provide more immediate and detailed feedback to MLLMs. It achieves this by breaking down complex visual reasoning into simpler Visual Question Answering (VQA) tasks, which offer dense reward signals. This approach helps models build foundational visual understanding skills from the ground up, acting as an effective “cold-start” for training.
Building on this, the team proposed REWARDMAP, a multi-stage reinforcement learning framework. REWARDMAP is specifically engineered to boost both the visual understanding and reasoning capabilities of MLLMs. It incorporates two main ideas to overcome the sparse reward problem and improve learning efficiency.
Difficulty-Aware Reward Design
The first core component of REWARDMAP is its smart reward system. Unlike traditional RL setups that might only give a simple “correct” or “incorrect” signal, REWARDMAP uses a more nuanced approach. It includes a “detail reward” that grants partial credit for getting parts of an answer right, such as identifying the correct origin or destination stops, or specific route names in a transit map. This helps guide the model even when it doesn’t get the full answer immediately. Additionally, the reward system is “difficulty-aware,” meaning it adjusts the reward based on how challenging the map or question is. This ensures that harder tasks receive appropriately scaled feedback, preventing the model from getting stuck or optimizing poorly due to overly sparse signals.
Also Read:
- Enhancing Multimodal Reasoning with Advanced Vision-Language Process Reward Models
- Geo-R1: A New Approach to Geospatial Reasoning in AI Models
Multi-Stage Reinforcement Learning
The second key innovation is a multi-stage RL training scheme. This approach moves away from conventional Supervised Fine-Tuning (SFT) by directly using RL from the start. The training is structured like a curriculum, progressing from easier perception-focused tasks to more complex reasoning challenges. This “easy-to-hard” progression, combined with the dense rewards from REASONMAP-PLUS, allows the model to gradually acquire and refine its skills. The training is divided into stages based on question type (e.g., binary judgment, counting, planning) and target task (visual understanding to visual reasoning). Within each stage, a touch of randomness in sample ordering prevents overfitting and enhances the model’s adaptability.
Experiments conducted on both REASONMAP and REASONMAP-PLUS datasets showed that each part of REWARDMAP contributes to better performance, and when combined, they yield the best results. Models trained with REWARDMAP demonstrated an average improvement of 3.47% across six different benchmarks, covering spatial reasoning, fine-grained visual reasoning, and general tasks beyond just transit maps. This indicates a significant enhancement in the models’ overall visual understanding and reasoning abilities.
Qualitative analysis further revealed that REWARDMAP helps MLLMs avoid common pitfalls like visual confusion and “hallucinations” (generating incorrect or repetitive information) when dealing with complex maps. The framework consistently produced correct routes, even on visually challenging maps.
In essence, REWARDMAP offers a robust solution to the challenges of sparse rewards in fine-grained visual reasoning for MLLMs. By integrating a detailed, difficulty-aware reward system with a structured multi-stage reinforcement learning curriculum, it enables models to learn more effectively and generalize better across various visual tasks. For more in-depth information, you can read the full research paper here.
