TLDR: A new framework automates the creation of scientific multiple-choice question-answering (MCQA) benchmarks from large paper corpora. A case study in radiation and cancer biology showed that using “reasoning traces” (distilled thought processes from GPT-4.1) as retrieval sources significantly and consistently improved the performance of small language models (1.1B-14B parameters), often enabling them to surpass GPT-4 on expert exams, outperforming direct retrieval from paper content.
The world of scientific knowledge is expanding at an incredible rate, making it challenging for artificial intelligence models to keep up with the latest discoveries. Traditional methods for evaluating these models, especially through multiple-choice question-answering (MCQA) benchmarks, often become outdated quickly or are too expensive and slow to create manually.
A new research paper introduces an innovative, scalable framework designed to automatically generate MCQA benchmarks directly from vast collections of scientific papers. This system aims to ensure that language models are continuously tested on current, diverse, and relevant scientific literature. The framework handles every step of the benchmark creation process, from parsing PDF documents and breaking them into meaningful sections, to generating questions and evaluating how well models perform.
As a practical demonstration, the researchers applied their framework to the field of radiation and cancer biology. They successfully generated over 16,000 multiple-choice questions from 22,000 open-access articles. This extensive dataset was then used to evaluate a range of small language models, varying in size from 1.1 billion to 14 billion parameters.
The evaluation compared three different approaches for these small models: a baseline performance without any additional help, performance with Retrieval-Augmented Generation (RAG) using semantic chunks directly from the papers, and RAG using “reasoning traces” distilled from a much larger and more capable model, GPT-4.1. Reasoning traces are essentially the step-by-step thought processes GPT-4.1 used to arrive at an answer, but without revealing the final answer itself.
The findings were significant: retrieving reasoning traces consistently led to better performance for the small language models. This improvement was observed on both the synthetically generated questions and on an expert-annotated benchmark, specifically the 2023 Astro Radiation and Cancer Biology exam. In some cases, several small models, when augmented with these reasoning traces, even managed to outperform GPT-4 on the Astro exam, despite GPT-4 being a much larger model.
This work highlights several key contributions. First, it provides a modular and scalable pipeline for creating MCQA benchmarks from scientific literature, designed to work efficiently on high-performance computing platforms. Second, it introduces a new benchmark of over 16,000 questions in radiation and cancer biology. Third, it offers a systematic evaluation of various small language models using RAG from both paper chunks and GPT-4.1 reasoning traces. Finally, the empirical results clearly show that reasoning trace retrieval significantly and consistently boosts the performance of small models in specialized domains, often more effectively than retrieving directly from the literature. This approach can even enable them to surpass the performance of larger models on expert assessments.
Also Read:
- Improving RAG Performance with Hierarchical Document Chunking
- HANRAG: Enhancing AI’s Question Answering Through Adaptive Retrieval and Noise Filtering
The researchers believe this framework can help keep benchmarks current with scientific progress and allow smaller, more efficient models to achieve top-tier performance in specialized scientific areas. It also offers a way to scale up the process of distilling reasoning from advanced Large Language Models to adapt Small Language Models, making them more capable for scientific tasks. The source code and benchmark artifacts are being released to support reproducible evaluations and future AI-for-science research. You can find more details about this research in the full paper available at Automated MCQA Benchmarking at Scale.
