TLDR: GuirlVG is a new reinforcement learning-based method for Graphical User Interface Visual Grounding (GUI-VG) that significantly outperforms traditional supervised fine-tuning (SFT) methods. It achieves state-of-the-art accuracy on GUI-VG benchmarks using drastically fewer training samples (e.g., 5.2K vs. 10M+), thanks to a systematic empirical study of reinforcement fine-tuning (RFT) components and a novel “Adversarial KL Factor” for stable training. This demonstrates RFT’s superior data efficiency and generalization capabilities for GUI agents.
Graphical User Interface Visual Grounding (GUI-VG) is a crucial ability for AI agents that interact with digital environments. Traditionally, this has been achieved by extensively fine-tuning multimodal large language models (MLLMs) using supervised learning. This approach, however, demands a lot of data and significant training costs.
As MLLMs become more advanced and even incorporate GUI data during their initial training, the need for such exhaustive post-training fine-tuning becomes less clear. This paper introduces a new method called GuirlVG, which uses reinforcement learning (RL) to improve GUI-VG more efficiently.
The researchers found that simply applying reinforcement fine-tuning (RFT) didn’t perform as well as the traditional supervised fine-tuning (SFT) methods. This led them to explore RFT in depth. They broke down RFT into its main parts and analyzed how each part could be optimized. A key innovation is the “Adversarial KL Factor,” a new technique that helps stabilize the training process by dynamically adjusting the penalty for deviating from the original model, preventing the model from over-optimizing for rewards.
Through extensive experiments, GuirlVG demonstrated remarkable efficiency. It achieved better performance than SFT methods that were trained on over 10 million samples, while GuirlVG itself only used 5.2 thousand training samples. This resulted in significant improvements: a +7.7% gain on ScreenSpot, a +17.2% gain on ScreenSpotPro, and 91.9% accuracy on ScreenSpotV2.
The Journey of GuirlVG’s Development
The development of GuirlVG was a step-by-step process, starting from a basic RFT setup and progressively adding improvements. Initially, the trivial RFT achieved 79.2% accuracy on ScreenSpot. By introducing a Soft Reward Function, which provides partial credit for correct output formats, the accuracy improved to 79.9%. Further enhancements included optimizing the prediction format to directly predict a point within the target bounding box, leading to 83.4% accuracy.
The Adversarial KL Factor proved to be a critical component, boosting accuracy to 87.4%. The researchers also found that using LoRA (Low-Rank Adaptation) for fine-tuning was significantly more efficient than full model fine-tuning, offering comparable performance with much less computational cost. Careful tuning of hyperparameters like group size and batch size also played a role in achieving optimal performance. Interestingly, the study revealed that withholding image resolution information during training but providing it at test time led to better spatial reasoning and improved accuracy.
Outperforming Previous Methods with Less Data
GuirlVG was rigorously compared against existing GUI-VG benchmarks. On ScreenSpot, it surpassed methods using hundreds of thousands to over ten million training examples, achieving superior accuracy with only 2,000 to 5,200 samples. For instance, it outperformed OS-Atlas by +7.0% to +7.7% in accuracy, despite OS-Atlas using thousands of times more data. This data efficiency suggests that RFT helps models generalize better, rather than just memorizing training data.
Similar trends were observed on ScreenSpot v2, where GuirlVG achieved a new state-of-the-art of 91.9% average accuracy, a +7.8% increase over OS-Atlas. On ScreenSpot-Pro, which focuses on high-resolution professional GUIs, GuirlVG showed strong generalization, achieving 36.1% accuracy, an absolute gain of +17.2% over the best SFT baseline. These results highlight the robustness and effectiveness of the RFT pipeline in complex real-world scenarios.
Also Read:
- SEAgent: An AI Framework for Autonomous Software Proficiency
- Understanding AI Assistants: A Deep Dive into OS Agents for Digital Device Control
Understanding GuirlVG’s Reasoning
Qualitative examples illustrate GuirlVG’s ability to not only achieve high accuracy but also provide interpretable reasoning steps. For instance, when asked to locate an Instagram icon, the model would reason about identifying social media icons, specifically camera icons, and then pinpoint the correct location. This explicit thinking process, enabled by the reinforcement learning approach, makes GuirlVG a reliable solution for GUI-VG tasks.
In conclusion, GuirlVG represents a significant step forward in GUI visual grounding. By systematically exploring and optimizing rule-based reinforcement fine-tuning, this research demonstrates that RFT can decisively outperform traditional supervised fine-tuning, offering a more data-efficient and generalizable alternative for developing GUI agents. You can read the full research paper for more details: GuirlVG Research Paper.
