TLDR: A study benchmarked GPT-4o, DeepSeek-V3, and GLM-4.5 as personalized learning assistants in a data structures tutoring scenario. Using Gemini as an AI evaluator, the research found GPT-4o consistently superior in providing clear, accurate, and actionable feedback, while DeepSeek-V3 showed moderate performance and GLM-4.5 lagged. The study highlights the potential of LLMs in education and proposes a robust evaluation framework.
Large Language Models (LLMs) are increasingly seen as powerful tools that can act as intelligent assistants, especially in personalized learning. However, there hasn’t been much systematic testing of how well these AI models perform in real-world educational settings. A recent study by Bo Yuan and Jiazi Hu addresses this gap by conducting a detailed comparison of three leading LLMs in a simulated tutoring task.
The research, titled “Benchmarking Large Language Models for Personalized Guidance in AI-Enhanced Learning,” aimed to evaluate GPT-4o, DeepSeek-V3, and GLM-4.5. The goal was to see how effectively these models could analyze a student’s quiz performance, understand their knowledge gaps, and then provide tailored guidance for improvement. You can read the full research paper here: Benchmarking Large Language Models for Personalized Guidance in AI-Enhanced Learning.
The Promise of Personalized Learning with AI
Personalized learning is an educational approach where teaching methods, resources, and pacing are adjusted to fit each student’s unique needs, abilities, and interests. This is a significant shift from the traditional “one-size-fits-all” model. Historically, providing individualized attention was difficult due to limited teacher time and resources. Assessments often focused only on correctness scores, missing deeper insights into student misconceptions. LLMs offer a groundbreaking opportunity to overcome these challenges by generating dynamic, context-aware, and individualized feedback in real-time.
Designing the Experiment
To simulate a realistic post-class tutoring scenario, the researchers created a dataset of quiz questions from an undergraduate data structures course. This dataset included questions, student answers, and correctness labels. The LLMs were not explicitly told the knowledge points for each question; instead, they had to identify these concepts themselves, adding a layer to their diagnostic test.
A crucial aspect of the study was the use of a unified prompt for all three LLMs. This prompt instructed each model to act as an intelligent tutoring assistant. Their tasks included: identifying key knowledge points for each question, inferring the student’s mastery level (mastered, partially understood, or not yet grasped), and generating personalized feedback. This feedback had to cover strengths and weaknesses, likely misconceptions, specific actionable learning strategies, and recommended resources.
Evaluating the AI Tutors
To ensure fairness and minimize human bias, Gemini 2.5 was employed as an independent virtual judge. Gemini’s role was to perform pairwise comparisons of the feedback generated by the tutoring LLMs. It evaluated the outputs based on five key pedagogical dimensions: accuracy of knowledge diagnosis, specificity and actionability of feedback, identification of misconceptions, instructional clarity, and appropriateness to the student’s level. Gemini provided a relative judgment, indicating which model was superior or if they were equally good.
Key Findings: GPT-4o Leads the Pack
The quantitative analysis, using pairwise win-tie-loss results and the Bradley-Terry model, established a clear performance hierarchy: GPT-4o consistently outperformed DeepSeek-V3, which in turn outperformed GLM-4.5. GPT-4o showed a statistically significant superiority in delivering personalized tutoring feedback.
A qualitative analysis further illuminated these differences:
- Clarity and Structure: GPT-4o consistently presented feedback in a clear, well-organized manner with headings and bullet points. DeepSeek-V3 was systematic but sometimes verbose, while GLM-4.5’s outputs were often narrative and fragmented.
- Depth of Knowledge Diagnosis: GPT-4o demonstrated strong diagnostic capabilities, linking errors to specific knowledge points and identifying common misconceptions. DeepSeek-V3 showed moderate depth, while GLM-4.5 often provided superficial feedback.
- Specificity and Actionability: GPT-4o offered concrete study strategies, including references to external platforms and specific coding exercises. DeepSeek-V3’s recommendations were less consistent and sometimes generic. GLM-4.5 typically produced very generic advice like “review definitions” or “practice more,” lacking personalization.
Also Read:
- Boosting Classroom LLMs: A Comparative Look at AI Retrieval for Accurate Learning
- AI Models Face Physics Olympiad Challenge: A New Benchmark Reveals Performance Gaps
Conclusion and Future Directions
The study concludes that LLMs hold genuine promise as AI teaching assistants capable of delivering personalized support. It emphasizes that evaluating such systems requires looking beyond just correctness; factors like clarity, diagnostic depth, actionability, and communicative tone are equally vital for educational value. While the study had limitations, such as a restricted dataset and a limited number of LLMs, it provides a robust methodological framework for future empirical research on LLMs in personalized learning. Future work could expand the scope to more diverse datasets, broaden evaluation dimensions to include fairness and long-term impact, and integrate human feedback for a more comprehensive assessment.
