TLDR: This research explores automated grading of students’ handwritten graphs in mathematics using multimodal meta-learning models and Vision-Large Language Models (VLLMs). Evaluating on a real-world dataset, the study found that meta-learning models excelled in simpler 2-way classification tasks, while VLLMs performed better in more complex 3-way classification. The research highlights the potential of AI in educational assessment, while also noting challenges such as data limitations, annotation consistency, and the practical applicability and cost of VLLMs.
As online education continues to grow, the need for efficient and consistent ways to grade student work, especially in mathematics, has become increasingly important. While automated grading systems have been widely used for text-based answers and mathematical expressions, there has been limited progress in automatically grading handwritten graphs, which are common in science, technology, engineering, and mathematics (STEM) courses.
A recent study tackles this challenge by implementing and comparing two advanced artificial intelligence approaches: multimodal meta-learning models and Vision-Large Language Models (VLLMs). The goal is to develop a system that can automatically grade handwritten graphs and accompanying text, offering teachers an objective and consistent second opinion.
The Challenge of Grading Handwritten Graphs
Traditional automated grading often relies on Natural Language Processing (NLP) for text. However, mathematical responses often combine text with complex symbols, formulas, and equations. While Optical Character Recognition (OCR) can convert handwritten math into text, visual elements like graphs, tables, and charts pose a unique problem that standard NLP or OCR cannot directly handle. This research specifically focuses on addressing this gap by developing models capable of assessing hand-drawn graphs.
How the Study Was Conducted
The researchers used a real-world dataset of anonymized images of hand-drawn graphs collected from a Swiss distance university’s online learning platform. Students created these graphs using digital canvases and Wacom tablets as part of economics-related mathematics courses. The dataset, initially comprising over 1,300 images, was filtered to 1,174 relevant images.
A crucial step involved data annotation. Since existing grades were inconsistent, an expert teacher collaborated with the researchers to define clear grading criteria (rubrics) for each task. These criteria were then used to systematically annotate the graphs, indicating which elements were correctly fulfilled. For example, a criterion might be “correct shift of demand curve to the right.”
Before feeding the images to the AI models, a semi-automatic data preprocessing pipeline was developed. This pipeline extracts the relevant graph regions from each image using computer vision techniques like contour detection. Any accompanying text within the image is extracted using Tesseract OCR. Both the extracted graph and text serve as inputs for the machine learning models. The grading system assigns a numerical grade based on the fulfillment of these criteria.
Meta-Learning vs. Vision-Large Language Models
The study evaluated two main types of AI models:
- Meta-Learning Models: These models are designed to learn from a small number of examples, making them suitable for situations where labeled data is scarce. They learn to adapt quickly to new tasks. The study tested several meta-learning algorithms, including Prototypical Network, FOMAML, ProtoFOMAML, Matching Network, and Relation Network. For graph analysis, a ResNet-18 architecture was used, and for text, a BERT model pre-trained on German text was employed. Both were fine-tuned during the learning process.
- Vision-Large Language Models (VLLMs): These are powerful AI models, such as variants of GPT-4, that have been trained on vast amounts of diverse data, including text and images. Unlike meta-learning, VLLMs use “in-context learning,” where they are provided with a few labeled examples as part of their input prompt and are expected to classify new queries without further fine-tuning. The study evaluated models like gpt-4.1, gpt-4o, and their mini/nano variants.
Key Findings
The results showed interesting differences in performance:
- For simpler tasks, specifically 2-way classification (where there are two possible grades), the best-performing meta-learning models, such as ProtoFOMAML and Prototypical Network, generally outperformed the VLLMs.
- However, for more complex tasks, specifically 3-way classification (where there are three possible grades), the top-performing VLLMs, like gpt-4o and o4-mini, slightly outperformed the meta-learning models.
An ablation study also confirmed that while graph representations are more informative for this task, incorporating text generally improves the model’s accuracy, indicating the benefit of a multi-modal approach.
Also Read:
- Unveiling BMMR: A New Benchmark for Multidisciplinary AI Reasoning
- MateInfoUB: A New Benchmark Reveals LLM Strengths and Weaknesses in Competitive Computer Science Education
Limitations and Future Directions
The study acknowledges several limitations. The graph annotations were performed by a single teacher, which could introduce bias. The dataset size and diversity were also limited, with some grade categories being highly imbalanced. Future work aims to expand the dataset, involve multiple expert annotators for more robust labeling, and explore more diverse prompting strategies for VLLMs.
Despite their promising results, VLLMs also presented practical challenges, including high computational costs and occasional inconsistencies when processing multiple test graphs in a single prompt. This highlights that while powerful, their reliability and practical applicability for automated grading still require further investigation.
Ultimately, the researchers envision integrating such a system into online learning platforms like Moodle. This could significantly enhance grading efficiency and consistency, providing educators with a valuable tool for initial evaluations or a second opinion on students’ graph-based responses. You can read the full research paper here.
