TLDR: A new research paper introduces FoRepBench, a benchmark dataset of 618 high-quality samples for training and evaluating AI models to automatically repair Excel formulas, especially runtime errors. The dataset is generated synthetically using a novel pipeline that leverages LLMs and a two-stage validation process, including execution checks and an LLM-as-a-Judge framework. This work also proposes a context-aware baseline repair technique and evaluates various LLMs, demonstrating the dataset’s utility and the cost-effectiveness of its generation.
Microsoft researchers have introduced a significant advancement in the realm of automated Excel formula repair with the unveiling of FoRepBench, a novel benchmark dataset designed to train and evaluate large language models (LLMs) in fixing complex spreadsheet errors. Excel, despite its widespread use, often presents a formidable challenge, particularly for those without a programming background, as logical mistakes or misunderstandings of functions can lead to frustrating runtime errors like #DIV/0! or #REF!.
While LLMs have shown promise in explaining these errors, the automatic correction of semantic runtime errors has remained an open problem. A major hurdle has been the severe lack of high-quality, comprehensive datasets for both training and rigorous evaluation of models in such scenarios. This new research directly addresses this gap.
The paper, titled “Benchmark Dataset Generation and Evaluation for Excel Formula Repair with LLMs,” details a unique approach to constructing this benchmark. The core of their methodology is a data generation pipeline that starts with a small collection of carefully curated seed samples from online forums. This pipeline then synthetically expands the dataset, leveraging few-shot prompting with LLMs. To ensure the quality and semantic accuracy of the generated data, it integrates a robust LLM-as-a-Judge validation framework, complemented by execution-based checks.
This meticulous process has yielded FoRepBench, a benchmark dataset comprising 618 high-quality samples. These samples cover common runtime errors, including #DIV/0!, #N/A, #NAME?, #REF!, and #VALUE!. Each entry in the dataset is rich with context, providing the tabular data, the faulty formula, its corrected version, and a natural language description of the user’s intent.
Beyond the dataset itself, the researchers also propose a context-aware baseline technique for Excel formula repair. This technique utilizes LLMs to consider both the faulty formula and relevant spreadsheet context, a crucial element often overlooked in prior work. They evaluated the performance of various LLMs, including GPT-4o, GPT-4.1, Phi-3, and Mistral, on this new benchmark using execution-based metrics. The analysis not only demonstrates the dataset’s quality through manual annotation but also offers valuable insights into error and function distributions.
The synthetic data generation pipeline, referred to as Bootstrap Generator, begins by curating a small set of high-quality seed samples from forums like MrExcel. These samples are meticulously filtered, parsed, and validated using Calc.ts, an Excel formula evaluation engine, to ensure they contain a faulty formula, correct formula, table context, and user query, all associated with a runtime error. This seed data then fuels the synthetic generation process, where one-shot prompting is used to create new, diverse samples. A two-stage validation process follows: first, Calc.ts verifies that the generated ‘correct’ formula executes without error and the ‘faulty’ one produces the expected runtime error; second, an LLM-as-a-Judge (LLM Validator) assesses the semantic alignment with user intent and assigns a difficulty level.
The evaluation of the baseline repair technique showed that larger models like GPT-4.1 and GPT-4o performed significantly better on FoRepBench compared to open-source models like Phi-3 and Mistral. Interestingly, the synthetic FoRepBench dataset proved to be relatively easier for the LLMs to repair compared to the manually curated seed dataset. This suggests that real-world errors, as found in the seed data, often involve deeper nesting and require more substantial semantic edits, while synthetic examples tend to be simpler, requiring fewer, more localized corrections. This highlights a potential area for future improvement in synthetic data generation to better capture real-world complexity.
The cost-effectiveness of the generation pipeline is also a notable finding. The creation of FoRepBench involved a total of 1,154 LLM API calls (59 for generation and 1,095 for validation), resulting in 618 high-quality samples at an average cost of approximately $0.02 per accepted sample. This demonstrates that large-scale, diverse datasets can be created with minimal manual effort, making the methodology highly scalable and adaptable to similar code repair tasks in other low-resource programming languages.
Also Read:
- Unlocking Code Repair: How Diffusion Models Learn to Fix and Generate Code
- Automating Spreadsheet Formatting with Predictive AI
The researchers acknowledge certain limitations, such as the LLM-based reviewer agent sometimes diverging from human judgment on contextual plausibility, and the narrow context windows of LLMs potentially omitting globally relevant spreadsheet information. Future work aims to address these by incorporating human-in-the-loop validation, refining reviewer agents, enhancing context selection, and extending the pipeline to multi-sheet and localized spreadsheets. FoRepBench is available as a resource for further research, and you can find more details in the full paper: Benchmark Dataset Generation and Evaluation for Excel Formula Repair with LLMs.
