TLDR: POSH is a new open-weight metric that uses scene graphs to guide LLMs-as-a-Judge for evaluating detailed image descriptions. It provides interpretable, fine-grained error localization and aggregate scores. Validated with DOCENT, a new benchmark of artwork with expert descriptions and human judgments, POSH shows stronger correlation with human ratings than existing metrics, including GPT4o, and is robust across image types. It also serves as an effective reward function for training VLMs.
As artificial intelligence continues to advance, particularly in its ability to describe images in intricate detail, a significant challenge has emerged: how do we accurately evaluate the quality of these descriptions? Traditional evaluation methods, often designed for shorter texts, struggle to assess the nuances of long, comprehensive image descriptions, especially when it comes to identifying subtle errors in attributes or relationships between objects.
A new research paper introduces a novel metric called POSH (PrOofing Scene grapHs) that aims to address this very problem. POSH provides a structured and interpretable way to guide large language models (LLMs) in judging the quality of detailed image descriptions. Unlike older metrics, POSH focuses on identifying fine-grained errors, such as mistakes in compositional understanding, and localizes these errors to specific parts of the text.
The core idea behind POSH is its use of “scene graphs.” Imagine a detailed map of an image that breaks down all its visual components: objects, their attributes (like color or size), and the relationships between them (like “man pouring water”). POSH extracts these scene graphs from both the AI-generated description and a human-written reference description. These graphs then act as structured rubrics, allowing an LLM to systematically compare the two descriptions. This process helps pinpoint exactly where the AI’s description might be inaccurate or incomplete.
The evaluation process in POSH involves three main steps. First, it extracts scene graphs from both the generated and reference descriptions. Second, it uses a question-answering approach with an open-weight LLM to check for the presence of each component from one scene graph in the other description. This granular scoring identifies specific mistakes (precision errors) and omissions (recall errors). Finally, these granular scores are aggregated to produce overall coarse scores for mistakes, omissions, and general quality, offering clear insights into the AI model’s performance.
To rigorously test POSH, the researchers also introduced a challenging new dataset called DOCENT. This benchmark is unique because it focuses on visual art, including paintings, sketches, and sculptures. It features 1,750 artworks, each paired with expert-written descriptions from the U.S. National Gallery of Art. What makes DOCENT particularly valuable are the human judgments it includes: art history students provided both granular (specific text span errors) and coarse (overall quality rankings) feedback on AI-generated descriptions. This rich dataset allows for a much deeper evaluation of image description metrics and the AI models themselves.
The findings show that POSH significantly outperforms existing metrics, including even advanced models like GPT4o-as-a-Judge, in correlating with human judgments on the DOCENT dataset. It demonstrated stronger correlations in identifying mistakes, omissions, and overall quality. Furthermore, POSH proved to be robust across different image types, maintaining its effectiveness on an existing dataset of web imagery called CapArena. The research also highlighted POSH’s potential as a reward function for training AI models, leading to better descriptions than traditional supervised fine-tuning methods.
- SSL4RL: Guiding AI to Deeper Visual Understanding with Self-Supervised Rewards
- Grasp Any Region: Advancing Multimodal AI for Detailed Visual Understanding
By introducing both POSH and DOCENT, this work establishes a new, demanding task for evaluating the progress of vision-language models. It extends detailed image description to the complex and socially impactful domain of assistive text generation for artwork, an area where current foundation models often struggle to achieve complete, error-free coverage. The researchers hope that these contributions will drive further advancements in creating more accessible and accurate image descriptions for everyone. You can read the full research paper here: POSH: Using Scene Graphs to Guide LLMs-as-a-Judge for Detailed Image Descriptions.
