TLDR: A study found that Generative AI (GenAI) performs at a first-class degree level in undergraduate mathematics exams, with significantly more consistent results than human students. This challenges the pedagogical relevance of traditional closed-book exams in uninvigilated, open-book settings with GenAI access, highlighting an urgent need to redesign assessments in mathematics to maintain pedagogical value and address potential inequalities.
Generative Artificial Intelligence (GenAI) tools, such as OpenAI’s ChatGPT, are rapidly changing the landscape of education, prompting a critical re-evaluation of how universities conduct assessments. As institutions explore alternatives to traditional in-person, closed-book examinations, concerns about academic integrity and the pedagogical value of assessments in uninvigilated settings with GenAI access have grown.
A recent study, titled “Evaluating undergraduate mathematics examinations in the era of generative AI: a curriculum-level case study,” by Benjamin J. Walker, Nikoleta Kalaydzhieva, Beatriz Navarro Lameda, and Ruth A. Reynolds, delves into this pressing issue. The research specifically investigates whether traditional closed-book mathematics examinations remain pedagogically relevant when hypothetically administered in uninvigilated, open-book environments where students could use GenAI tools.
Study Methodology
To address their research question, the team adopted an empirical approach. They generated, transcribed, and blind-marked GenAI submissions to eight undergraduate mathematics examinations at a Russell Group university in the UK. These exams covered the entire first-year mathematics curriculum for the 2024–2025 academic year. The GenAI tool used was OpenAI’s ChatGPT, specifically the chatgpt-4o-latest model, accessed via its API in March 2025. Each question was converted to plain text, and a specific prompt was used to instruct the AI to respond as a mathematics undergraduate student taking an exam at a UK university, emphasizing precision, thoroughness, and conciseness. Eight responses were requested for each question, with the first five complete responses used for marking. These GenAI responses were then transcribed by hand into exam scripts, scanned, and submitted to the digital marking platform alongside actual student scripts, ensuring markers were unaware of the AI submissions.
The researchers combined independent GenAI responses to individual questions to simulate a large number of possible exam entries, allowing for a meaningful evaluation of GenAI performance at both the module and curriculum levels. This comprehensive approach enabled a direct comparison between GenAI and student attainment.
Key Findings: GenAI Performance Across the Curriculum
The study revealed striking insights into GenAI’s capabilities. Overall, GenAI achieved an average mark of 71.1 out of 100 across the first-year curriculum, which is equivalent to a low first-class degree. This average was statistically indistinguishable from the student cohort’s mean of 69.1. While the average performance was similar, the consistency of GenAI was remarkably higher. The GenAI cohort exhibited a significantly lower variance in scores, with a standard deviation of 1.5 compared to the student cohort’s 15.1. This suggests that GenAI performance is much more consistent than that of human students in invigilated examinations.
Module-Specific Variability
Despite the strong overall performance, GenAI attainment varied significantly between different modules. For instance, GenAI substantially outperformed students in Multivariable Calculus, partly due to a programming question where it provided nearly perfect solutions. This strong performance in programming-related tasks is perhaps unsurprising given the prevalence of GenAI in software engineering.
Conversely, GenAI’s attainment in Integration was relatively low. A detailed inspection of the marked scripts showed that GenAI frequently assumed results that were beyond the scope of a first-year course, often using concepts typically taught in the third year of a mathematics undergraduate program. This highlights the importance of providing sufficient context to GenAI tools, as no module-specific information or lecture notes were given to the AI.
Question-Level Analysis: Strengths and Weaknesses
A deeper dive into individual questions identified recurring themes in GenAI’s performance:
- Procedural Tasks: GenAI generally performed well in procedural tasks, such as solving quadratic equations for eigenvalues, correctly executing intermediate algebraic steps. However, it often made errors in the final numerical computation, though these minor inaccuracies had little impact on overall grades.
- Structured Multi-Part Questions: In questions designed to scaffold solutions, GenAI sometimes struggled to connect earlier results to subsequent parts. For example, it might correctly solve an initial part but then misapply a theorem or repeat a calculation with an error in a later, related part.
- “Prove or Disprove” Questions: GenAI’s performance was unreliable in “prove or disprove” questions, where it had to decide the truth of a statement and provide a proof or counterexample. It sometimes incorrectly “proved” false statements or provided incorrect counterexamples.
- Following Specific Instructions: GenAI occasionally failed to follow explicit instructions, such as rewriting a formula without using certain notation or quantifiers, continuing to use the prohibited constructs.
- Questions with Sketches/Graphs: Surprisingly, GenAI performed well on questions requiring sketches or graphs, providing comprehensible instructions that a user could translate into correct diagrams, despite being limited to text-only prompts and answers.
Also Read:
- Navigating Generative AI in Advanced University Mathematics: A Pilot Study’s Insights
- AI Assists in Grading Handwritten Chemistry Exams: A Hybrid Approach for Efficiency and Accuracy
Implications for Assessment Design
The study concludes that today’s closed-book mathematics examinations would likely lose their pedagogical relevance if used as uninvigilated assessments with GenAI access. GenAI’s ability to perform at a first-class degree level, combined with its significantly lower variance in scores, means these assessments would do little to measure students’ alignment with intended learning outcomes or meaningfully distinguish learners from one another. The researchers emphasize that improvements in GenAI capabilities will only strengthen these conclusions, making the need for assessment redesign even more urgent.
The findings underscore the necessity for higher education institutions to redesign mathematics assessments for unsupervised settings. This redesign is crucial to maintain pedagogical value and address the potential for GenAI to exacerbate socioeconomic inequalities, as access to more advanced, paid GenAI tools could further impact results.
