TLDR: RECODE is an AI framework that improves how multimodal large language models (MLLMs) understand complex visuals like charts and diagrams. It works by “derendering” images into executable code, which allows the AI to precisely reconstruct and verify the visual information. This agentic system generates multiple code candidates, selects the best one using a “critic,” and then iteratively refines the code to reduce errors. This process enables more accurate calculations and logical inferences, significantly boosting performance on visual question-answering tasks across various benchmarks.
Multimodal Large Language Models (MLLMs) have made significant strides in understanding visual information, from describing scenes to recognizing object relationships. However, these advanced AI systems often encounter difficulties when tasked with precise reasoning over structured visuals such as charts, graphs, and diagrams. The challenge lies in the fact that pixel-based perception alone lacks a robust mechanism for verifying the accuracy of its interpretations, leading to potential errors in complex analytical tasks.
To address this critical gap, researchers have introduced RECODE, an innovative agentic framework designed to enhance visual reasoning through code generation. RECODE leverages a novel approach called “derendering,” which essentially involves reverse-engineering a visual input into executable code. This process transforms an ambiguous perceptual task into a verifiable, symbolic problem, enabling the AI to perform precise calculations and logical inferences.
How RECODE Works: An Agentic Approach
The RECODE framework operates through a closed-loop system of generation, critique, and refinement. Here’s a breakdown of its key steps:
1. Multi-Candidate Code Generation: Initially, the RECODE agent generates several candidate programs that could plausibly reproduce the input image. To improve the quality of this initial code, the system employs a hierarchical task decomposition strategy, breaking down the complex derendering task into smaller, manageable parts. It also incorporates a determinism constraint, ensuring the generated code accurately reflects observed data values rather than producing stochastic outputs. Furthermore, Optical Character Recognition (OCR) is integrated to extract and utilize textual information—like labels, titles, and units—from the image, providing crucial context for code generation.
2. Candidate Selection via Critic: Once multiple code candidates are generated, each program is executed to render a corresponding image. A “critic” function then evaluates the similarity between these rendered images and the original input. Simple yet effective pixel-based metrics, such as Mean Squared Error (MSE), have been found to be highly efficient and accurate in identifying the most faithful reconstruction. The code corresponding to the highest-scoring rendered image is selected as the best representation of the agent’s current understanding.
3. Iterative Self-Refinement: Starting with the best candidate code, the agent enters a refinement loop. It is prompted to analyze both the original image and its own re-rendered image, explicitly identifying any discrepancies. Based on these identified errors, the agent autonomously revises its code. This iterative process allows RECODE to progressively improve the fidelity of its generated code, capturing fine-grained details and correcting errors that a single-pass generation might miss. This loop can be repeated multiple times, leading to increasingly accurate code representations.
4. Answer Question: After the refinement rounds, the agent utilizes the final, high-fidelity code, along with the original image, to answer the given natural language query. The symbolic representation provided by the code enables the agent to perform complex reasoning tasks with greater accuracy and verifiability.
Also Read:
- Enhancing Mathematical Reasoning with Code-Driven Visual Thinking
- Improving Multimodal AI with Human-Like Visual Reasoning
Significant Performance Gains
RECODE has demonstrated impressive results across various visual reasoning benchmarks. On the CharXiv-Reasoning dataset, it achieved a 77% accuracy after two rounds of refinement, marking a substantial 19% absolute improvement over baseline models that do not use derendering. For the ChartQA dataset, RECODE set a new state-of-the-art performance with 93.2% accuracy, even surpassing chart-pretrained foundation models. Its capabilities also extend to formal diagrammatic reasoning, showing strong results on the Geometry3K benchmark with 94.2% accuracy.
These results underscore the power of grounding visual perception in executable code. By transforming ambiguous pixel information into precise, verifiable code, RECODE offers a new pathway toward more accurate and reliable multimodal reasoning for AI systems. For more details, you can read the full research paper here.
