TLDR: DUET is a new LLM-based tool that provides automated, structured feedback on student-generated UML and ER diagrams. It converts diagrams to text, compares them against a reference solution, and generates personalized hints for students and insights for educators. Evaluated through interviews, DUET offers benefits like scalability and instant feedback but also raises concerns about reliability and potential misuse, emphasizing its role as a supplementary learning aid rather than an assessment tool.
In the realm of computer science education, Unified Modeling Language (UML) and Entity-Relationship (ER) diagrams are fundamental tools for teaching software design and system analysis. However, students often grapple with the complexities of these diagrams, requiring abstract thinking, contextual understanding, and mastery of both syntax and semantics. Traditional teaching methods frequently fall short in providing scalable, personalized feedback, especially in large classes, leaving a significant gap in student support.
Addressing this challenge, researchers have introduced DUET (Diagrammatic UML & ER Tutor), a pioneering prototype of an AI-powered tool designed to automate feedback on student-generated UML and ER diagrams. DUET aims to foster self-directed learning and provide educators with analytical insights to refine their instructional strategies. You can learn more about the research behind DUET by reading the full paper here: Automated Feedback on Student-Generated UML and ER Diagrams Using Large Language Models.
How DUET Works: A Multi-Stage AI Pipeline
DUET operates through a sophisticated multi-stage Large Language Model (LLM) pipeline. The process begins when a student uploads their diagram, which can be either hand-drawn or digitally created. Simultaneously, an instructor provides a reference diagram.
The first crucial step involves diagram recognition and conversion. OpenAI’s GPT-4o, a powerful LLM, takes the uploaded diagram images and converts them into a textual representation known as PlantUML code. This conversion is vital because LLMs are currently more adept at processing text than extracting high-detail information from images.
Once both the student’s and the reference diagrams are in PlantUML format, a smaller, specialized LLM, such as Mistral-7B-Instruct-v0.3, steps in. This LLM meticulously compares the two textual representations, identifying structural differences like missing or additional classes, modified attributes, or changes in relationships. This comparison focuses purely on objective discrepancies without making judgments.
Finally, based on these identified differences, the smaller LLM generates two types of structured feedback. For students, it provides reflective, neutral hints that encourage them to re-examine specific aspects of their diagram without explicitly labeling elements as correct or incorrect. For educators, the tool offers valuable insights into common student misunderstandings and misconceptions, which can inform future teaching adjustments. All feedback is organized into predefined categories like classes, attributes, and relationships, presented in a clear, structured Markdown format.
Evaluation and Insights
To evaluate DUET, the researchers conducted semi-structured interviews with six participants, including two university lecturers and four teaching assistants (TAs) who regularly work with UML and ER diagrams. The evaluation highlighted several key benefits and limitations.
Participants praised DUET’s ability to provide instant and iterative feedback, offering a personalized, real-time learning aid. This allows students to correct errors and deepen their understanding at their own pace, unconstrained by the availability of instructors or TAs. Its scalability was also a significant advantage, making it suitable for large courses without increasing the workload for teaching staff. Some even saw the potential for reducing TA workload and ensuring fairer assessments through standardized feedback.
However, concerns were also raised. The non-deterministic output of LLMs, leading to potentially variable or inaccurate feedback across different executions, was a major point of apprehension, as it could confuse students. The possibility of misuse, where students might generate solutions rather than developing them independently, was also identified. Participants emphasized that DUET should serve as a learning tool to improve and learn interactively, rather than an automatic assessment tool, and should not replace human TAs but rather supplement their efforts.
Also Read:
- Unpacking AI’s Role in Software Security: A Deep Dive into Vulnerability Detection
- Guiding Self-Taught Programmers: A Storytelling Approach to Learning
Future Directions
The DUET prototype represents a promising step towards integrating LLMs into modeling education. Future work will focus on improving the tool’s accuracy, refining feedback presentation, and adding features like bulk upload functionality and interactive clarification. Measures to mitigate misuse, such as preventing direct solution generation, will also be explored. Ultimately, DUET aims to support reflective learning and iterative improvement in introductory modeling courses, helping students engage more deeply with core concepts and allowing educators to identify common misconceptions more systematically.
