TLDR: This research introduces Property Tracing (PT), a novel approach that combines Model Tracing (MT) and Constraint-Based Modeling (CBM) to accurately diagnose student problem-solving strategies in stepwise tasks. PT effectively identifies correct student steps even when multiple actions are combined, a common challenge for traditional MT systems. Evaluated on a dataset of quadratic equation solutions, PT’s diagnoses showed perfect alignment with human teacher assessments, demonstrating its potential to provide more precise and helpful feedback in educational software.
Assessing how students solve problems step-by-step in educational software has always been a challenge. Two main approaches, Model Tracing (MT) and Constraint-Based Modeling (CBM), have been widely used to diagnose student input in these stepwise tasks. Model Tracing excels at tracking consecutive steps a student takes, essentially following a predefined ‘expert’ path. However, it often struggles when students combine multiple steps into one, even if their combined action is mathematically correct. This often leads to a generic ‘Correct rewrite step, but unknown’ diagnosis, which isn’t very helpful for students.
Constraint-Based Modeling, on the other hand, is more flexible. It checks if the student’s input satisfies certain properties or ‘constraints,’ allowing for diagnosis even when steps are combined. However, CBM tends to have lower ‘cognitive fidelity,’ meaning it might not align as closely with how a human expert would reason through the problem.
Researchers Gerben van der Hoek, Johan Jeuring, and Rogier Bos from Utrecht University have proposed a novel approach called Property Tracing (PT) that merges the strengths of both MT and CBM. Their goal was to design a system that could provide accurate strategy diagnoses even when students combine several steps, addressing the limitations of existing methods.
How Property Tracing Works
The core idea behind Property Tracing is to identify variations in a student’s solution compared to an ideal model solution by using constraints. Instead of relying solely on a strict sequence of production rules (like MT), PT defines constraints as properties that a student’s input should share with a step of the expert strategy. This allows the system to understand if a student is following the correct strategy, even if they’ve taken a shortcut or combined steps.
The PT approach uses ‘normal forms’ to compare different representations of an object (like an equation) and ‘relations’ to check specific properties. For example, when solving quadratic equations, PT might check if two equations have the same simplified form, the same number of terms, or if they are both derived to zero. The order in which these relations are checked is crucial, as it helps provide more relevant feedback.
Evaluation and Results
To validate their approach, the researchers applied PT to an existing dataset of student steps from a study on solving quadratic equations, originally collected in 2010. This dataset included over 2000 instances where the traditional IDEAS domain reasoner (an MT variant) had diagnosed student steps as ‘unknown’ because multiple steps were combined.
The PT system was able to process these diagnoses quickly, averaging 0.17 seconds per diagnosis. Out of 2048 student steps, PT diagnosed 1749 as correct strategy applications where steps were combined, and the remaining 299 as strategy deviations. To assess the accuracy, two experienced mathematics teachers independently coded a random sample of 140 unique steps (70 deviations and 70 applications). The results were highly encouraging: the PT system’s diagnoses aligned perfectly with the teachers’ coding for all 140 student steps. This remarkable alignment demonstrates the effectiveness of the PT design.
Also Read:
- A New Framework for Temporal Reasoning with Constraints in Answer Set Programming
- KROMA: Enhancing Ontology Matching with Context-Aware Language Models
Implications and Future Directions
The Property Tracing design offers a balanced solution, providing high cognitive fidelity while remaining computationally feasible. It can offer specific strategy hints when a student follows the desired strategy and provide targeted feedback based on violated relations when a student deviates. This detailed diagnosis can also contribute to building a more accurate student model, tracking their understanding and application of strategies.
The researchers argue that the PT approach is generalizable to various domains, not just mathematics. The underlying principles of defining expert strategies, identifying variations through normal forms, and describing variations with ordered relations can be applied to other problem-solving procedures. While this study validated the accuracy of PT diagnoses, future work will focus on classroom experiments to see if students actually benefit from the feedback provided by PT. You can read the full research paper for more details: Combining model tracing and constraint-based modeling for multistep strategy diagnoses.
