TLDR: A new research paper introduces EDUMATH, a system that uses large language models (LLMs) to generate customized, standards-aligned math word problems (MWPs) for K-12 students. The project developed the first teacher-annotated dataset (STEM) for this task and trained EDUMATH models that achieve state-of-the-art performance. A study with grade school students found they performed equally well on LLM-generated and human-written MWPs, but overwhelmingly preferred the AI-generated problems due to their personalized topics, demonstrating LLMs’ potential to enhance math education and reduce teacher burden.
Math word problems (MWPs) are a cornerstone of K-12 education, serving as vital tools to assess students’ understanding of mathematical concepts. However, teachers often face a significant challenge: customizing these problems to align with individual student interests and ability levels. This customization is crucial for boosting engagement and learning outcomes, but with large class sizes and increasing teacher burnout, finding the time to personalize MWPs for every student is nearly impossible.
A new research paper, EDUMATH: Generating Standards-aligned Educational Math Word Problems, proposes an innovative solution: leveraging large language models (LLMs) to automatically generate MWPs that are tailored to student interests and adhere to specific math education standards. This approach aims to alleviate the burden on teachers while enhancing the learning experience for students.
The Need for Personalized Math Problems
Personalizing MWPs can significantly improve student interest in math and ensure that problems are at an appropriate readiness level. Traditionally, teachers manually create or adapt these problems. However, existing resources like educational websites or practice tests often lack customization options and are not easily searchable by specific math standards or topics. While LLMs show promise in natural language understanding and math reasoning, previous studies have indicated their struggles in generating high-quality educational MWPs, highlighting the need for further research and refinement.
EDUMATH’s Comprehensive Approach
The EDUMATH project addresses this gap by focusing on generating educational MWPs that are not only customized but also standards-aligned and come with clear, natural language solutions. To achieve this, the researchers developed a rigorous evaluation framework based on four key criteria:
- Solvability: Ensures the problem can be solved with the given information and presents a mathematically possible scenario.
- Accuracy: Verifies that the generated solution is correct, including both the final answer and the intermediate reasoning, which must be clear and readable for students and teachers.
- Educational Appropriateness: Assesses if the problem’s content, grammar, and context are suitable for 3rd-5th grade students in a school setting, avoiding confusing or inappropriate material.
- Standards Alignment: Confirms that the MWP effectively incorporates the pre-specified math standard, including specific topics and complexity constraints.
A unique aspect of EDUMATH’s methodology is the involvement of human experts, specifically real teachers, in the evaluation process. This human-expert and LLM-judge approach was used to evaluate over 11,000 MWPs, ensuring the nuanced understanding required for educational content.
Introducing the STEM Dataset
A significant contribution of this research is the creation of the Standards-Targeted Educational Math (STEM) dataset. This is the first teacher-annotated dataset specifically designed for standards-aligned educational MWP generation. It comprises over 2,500 MWPs, each with gold labels for Virginia Standards of Learning (VA SOLs) and step-by-step solutions crafted to be readable for students and teachers. Unlike existing datasets primarily focused on math reasoning, STEM prioritizes high-quality grammar, clear question-answer pairs, and alignment with specific grade-level standards, making it ideal for training advanced MWP generators.
Developing the EDUMATH Models
Using the STEM dataset, the researchers finetuned two open large language models: EDUMATH 12B and EDUMATH 30B. These models underwent supervised finetuning and Kahneman-Tversky Optimization, further enhanced by a trained text classifier to filter for high-quality outputs. The training demonstrated the significant value of the annotated data, as each stage of finetuning led to a statistically significant improvement in the models’ ability to generate high-quality, standards-aligned MWPs.
Impressive Performance and Student Preference
The EDUMATH models achieved state-of-the-art performance in generating standards-aligned educational MWPs. EDUMATH 30B notably outperformed all open and closed baseline models, including larger and more capable ones, in meeting all four evaluation criteria. EDUMATH 12B also showed strong performance, nearly matching larger models. The generated MWPs were found to be of high quality, with low perplexity scores, and semantically similar to human-written problems, indicating their diversity and authenticity.
Perhaps the most compelling finding comes from a study involving grade school students. Third to fifth graders were given worksheets with both human-written and EDUMATH-generated MWPs, customized to their interests. The results showed that students performed similarly on both types of problems, confirming the human-quality of the LLM-generated content. Crucially, students consistently preferred the LLM-generated MWPs, often citing their interest in the problem’s topic as the primary reason. This preference was even stronger when customization was at the individual student level.
Also Read:
- Evaluating Multimodal AI for Grading Handwritten Student Math Work
- Unlocking How Large Language Models Perceive Problem Difficulty
A Future for AI in Education
The EDUMATH project offers strong evidence that large language models can significantly contribute to K-12 math education. By automatically generating customized, standards-aligned math word problems, LLMs can reduce teacher workload and provide students with engaging, personalized practice materials. This research paves the way for future developments, including expanding to other grade levels and exploring multimodal math problems that incorporate images or tables. For more details, you can read the full research paper here.
Authors: Bryan R. Christ, Penelope Molitz, Jonathan Kropko, Thomas Hartvigsen
