TLDR: The research paper introduces ICRAG, a framework that converts natural language tasks into executable Python programs through iterative refinement and external knowledge retrieval. It addresses the ‘black box’ problem of LLMs by providing explicit, verifiable reasoning in code. Evaluated across 13 diverse benchmarks, ICRAG achieved up to 161.1% relative improvement, particularly in legal and medical domains. The study also analyzes the unique complexity of code generated from natural language tasks, highlighting its potential for transparent and auditable AI solutions.
A new research paper titled “Codifying Natural Language Tasks” introduces an innovative framework called ICRAG, which aims to bridge the gap between human language and executable computer code. Traditionally, text-to-code techniques have focused on generating code from prompts that are already intended to be in code form. However, many real-world problems, such as legal judgments or medical diagnoses, are expressed in natural language but require precise, verifiable reasoning.
The authors, Haoyang Chen and Kumiko Tanaka-Ishii, highlight a significant challenge with current large language models (LLMs): while they can process natural language, their reasoning often remains a “black box,” lacking transparency and verifiable correctness. This is particularly problematic for high-stakes domains where trust and accountability are crucial.
ICRAG, which stands for Iterative Co-refinement with Retrieval-Augmented Generation, tackles this by transforming natural language tasks into executable programs. Unlike previous methods that often rely solely on an LLM’s internal knowledge, ICRAG enhances this process by iteratively refining the generated code using external knowledge. This external knowledge comes from two main sources: domain-specific resources (like legal statutes or medical textbooks) and similar solved examples from GitHub.
The framework operates through a continuous loop: an initial program is generated from the natural language task. If this program encounters an issue during execution (like an undefined function), ICRAG queries the LLM to simulate the execution result. Simultaneously, it retrieves relevant external information, which could be non-code text (e.g., a legal rule) or code snippets. This retrieved evidence is then used to revise and improve the program, creating a robust “retrieval-and-repair” cycle. This iterative process ensures that the program is grounded in explicit knowledge, reducing the chances of factual inaccuracies or “hallucinations” by the LLM.
The researchers evaluated ICRAG across 13 diverse benchmarks, including mathematical reasoning, logical inference, causal judgment, legal reasoning, medical QA, and challenging Big-Bench tasks. The results were impressive, with ICRAG achieving an average of 32.7% higher accuracy compared to strong text-based baselines. For instance, in legal tasks like CAIL, accuracy jumped from 51% to 69%, and in medical QA (PubMedQA), it soared from 32% to 77%. These significant gains underscore the effectiveness of translating complex natural language reasoning into a verifiable code format.
A key finding was the efficiency of ICRAG’s iterative refinement; most tasks converged to a solution within three retrieval iterations. This demonstrates that the framework can quickly identify and incorporate missing pieces of information, combining logical and factual correctness.
Beyond performance, the study also delves into the nature of the generated code, termed “computables.” An analysis of these programs revealed that code derived from natural language tasks often exhibits higher complexity and unique patterns compared to generic human-written Python code. For example, legal and causal reasoning tasks frequently led to deeply nested logical conditions and complex combinations of reasoning skills. This insight helps identify areas where current LLMs still struggle and where future research can focus on improving coding capabilities for complex language problems.
The distinction between “generalizable” and “non-generalizable” tasks was also explored. For generalizable tasks (like math problems), the generated code proved reliable and correct when applied to new, valid inputs. For non-generalizable tasks (like specific legal cases), the code faithfully encoded the logic for that particular instance, with correctness aligning closely with the initial execution accuracy.
The impact of external knowledge was further examined, showing that while a larger pool of in-domain knowledge consistently improved accuracy, even injecting unrelated GitHub code could offer some benefit by providing reusable algorithmic structures. However, in-domain knowledge remained crucial for optimal performance.
Also Read:
- Guiding Language Models for Better Tool Use and Clearer Decisions
- Enhancing Facility Layout Problem Solving with AI-Powered Knowledge Graphs
In conclusion, ICRAG represents a promising direction for AI systems, enabling them to not only reason through complex natural language problems but also to construct transparent, auditable, and verifiable code solutions. This work paves the way for a future where AI can compile natural language specifications into reliable software. You can read the full paper here: Codifying Natural Language Tasks.
