TLDR: A new study reveals that AI models, particularly those employing supervised learning with Large Language Model (LLM) uncertainty, are more accurate than university professors at estimating the difficulty of True/False exam questions in Machine Learning and Neural Networks. This suggests AI can significantly enhance the quality and consistency of educational assessments.
Designing fair and effective exams is a complex task, and one of the biggest challenges is accurately gauging how difficult each question will be for students. Traditionally, professors rely on their expertise to estimate question difficulty, but a recent study suggests that Artificial Intelligence (AI) methods, particularly those based on Natural Language Processing (NLP), might be significantly better at this crucial task.
The research, titled “NLP Methods May Actually Be Better Than Professors at Estimating Question Difficulty”, compared the performance of three university professors with various Large Language Model (LLM)-based approaches. The goal was to see who could more accurately predict the percentage of students who would answer True/False exam questions correctly in subjects like Neural Networks and Machine Learning.
The study used a dataset of real exam questions from the University of Groningen. For each question, the actual percentage of students who answered correctly was known, serving as the ‘ground truth’ for difficulty. Three professors, experts in the field but not having taught these specific courses, were asked to estimate these percentages. They were given one example question with its true difficulty to help calibrate their estimates.
On the AI side, two main types of NLP methods were tested. The first involved directly asking powerful LLMs, specifically Gemini 2.5 and Gemini 2.0, to estimate the question difficulty. These models were prompted in two ways: by evaluating each question individually or by considering the entire set of questions at once, similar to how professors would review an exam.
The second, more advanced AI approach, used a technique called ‘LLM Uncertainty’. Here, a set of nine different LLMs were prompted to solve each question, and their ‘uncertainty’ in providing an answer was used as a feature. This uncertainty, along with other text-based features, was then fed into supervised learning models (like Support Vector Machines, Random Forest, and Linear Regression) to predict the question difficulty. This method required a small amount of training data from previous exams.
The results were striking. The professors showed a limited ability to distinguish between easy and difficult questions, with their estimates often not correlating well with actual student performance. There was no consistent pattern of over- or underestimation, but overall, their predictions were not highly accurate.
Directly prompting LLMs, especially Gemini 2.5, performed better than the professors. Gemini 2.5 consistently showed higher accuracy and better correlation with the true question difficulties compared to the human experts. Asking the LLM to evaluate questions one by one generally yielded better results than having it assess the entire set at once.
However, the most impressive performance came from the supervised learning method that utilized LLM uncertainty. Even with a relatively small training dataset (as few as 42 samples), this approach significantly outperformed both the professors and the direct LLM prompting. The Support Vector Machine model, trained on LLM uncertainties, achieved the highest accuracy and strongest correlation with the actual question difficulties.
This research highlights that while professors are indispensable for many aspects of exam design and education, AI can provide valuable assistance in accurately estimating question difficulty. The findings are particularly significant because they extend to questions involving complex mathematical notation and reasoning, a domain where previous NLP successes were less common. The study suggests that universities could leverage archived exam data to train such AI systems, leading to more consistent and well-designed assessments.
Also Read:
- AI Models Show Promise in Boosting Environmental Knowledge
- Understanding AI Grading Uncertainty with Semantic Entropy
While the best-performing AI system requires some computational resources and training data, the study also notes that simply using a state-of-the-art LLM directly can still offer better performance than human estimation, and is much simpler to implement. The authors emphasize that any such AI system should be used in a ‘human-in-the-loop’ fashion, where professors can still review and adjust AI-generated estimates, especially for nuanced cases where AI might lack broader context. For more details, you can read the full paper here.
