TLDR: Next Token Knowledge Tracing (NTKT) is a novel approach that redefines how student knowledge is modeled by reframing the task as a next-token prediction problem using pretrained Large Language Models (LLMs). Unlike traditional methods that often ignore question text, NTKT integrates full question content and student interaction histories as text sequences. This method significantly improves predictive accuracy over existing models and demonstrates superior generalization capabilities in cold-start scenarios involving new students or unseen questions, highlighting the importance of linguistic context in understanding student behavior.
Understanding how students learn and predicting their future performance is a fundamental challenge in education, especially with the rise of AI-powered learning tools. This field, known as Knowledge Tracing (KT), traditionally relies on simple data like whether a student answered correctly, along with basic metadata such as skill tags. However, a significant piece of the puzzle has often been overlooked: the actual text of the educational questions themselves.
Many existing Knowledge Tracing models operate on what’s called an “ID-based approach.” This means they use numerical identifiers for questions and concepts, often requiring extensive manual annotation by experts. This method frequently discards the rich pedagogical insights embedded within the question text, limiting the models’ ability to accurately predict student responses and generalize to new, unseen questions or students. Imagine trying to understand a student’s grasp of a subject without ever seeing the questions they’ve been asked – much of the context and nuance is lost.
Introducing Next Token Knowledge Tracing (NTKT)
A novel approach called Next Token Knowledge Tracing (NTKT) redefines this task by leveraging the power of pretrained Large Language Models (LLMs). Instead of treating questions as mere IDs, NTKT reframes Knowledge Tracing as a “next-token prediction” problem. This means that both a student’s past interactions and the content of the questions are represented as sequences of text. By doing so, LLMs can learn intricate patterns in both student behavior and the language of the questions, leading to a more comprehensive understanding of student knowledge.
The NTKT framework involves two main phases: data preparation and fine-tuning. In data preparation, student interaction histories – including the questions asked, answer options, and whether the student answered correctly – are converted into natural language sequences. These sequences are structured using XML-style tags, similar to how prompts are engineered for LLMs, to help the model distinguish between different parts of the information. For example, a question might be enclosed in <Q> </Q> tags, and the correctness of an answer in <cr> </cr> tags.
For fine-tuning, NTKT uses efficient adaptation techniques like Low-Rank Adaptation (LoRA) to train LLMs. The model is trained to predict the next token in the sequence, which in this case, is the student’s response (e.g., “Correct” or “Incorrect”). A clever aspect of NTKT is its “selective loss masking,” where the model only calculates its prediction error on the tokens that indicate the answer outcome, ignoring other tokens. This focuses the learning process directly on predicting correctness, while still allowing the LLM to use the full context of the question and history.
Significant Performance Improvements
Experiments conducted with NTKT demonstrated substantial improvements over state-of-the-art neural Knowledge Tracing models. Using the Eedi dataset, a large collection of multiple-choice mathematics problems, NTKT consistently achieved higher scores across key metrics like F1-score, Accuracy, and AUC. For instance, the LLaMA 3B NTKT model showed an F1-score of 90.20%, significantly outperforming traditional baselines that were in the 70s. Even the smallest NTKT model (1B parameters) surpassed all traditional baselines, highlighting the efficiency and effectiveness of leveraging pretrained LLM representations.
A crucial finding was the impact of including the full question text. Ablation studies showed that models performed best when provided with the complete question text, confirming that natural language content offers vital pedagogical cues. Models that only received question IDs or concept tags showed a decline in performance, especially in the reliability of their probability estimates.
Furthermore, NTKT proved to be much better at handling “cold-start” scenarios – situations where models encounter new students or entirely new questions they haven’t seen during training. In user cold-start scenarios, NTKT models quickly adapted to new learners, achieving higher F1 scores from the very first interaction. For question cold-start, NTKT maintained stable performance on unseen questions, unlike baseline models which showed a noticeable drop. This indicates that NTKT learns generalizable representations rather than simply memorizing identifiers.
Also Read:
- MiRAGE: A New Framework for Detecting Student Misconceptions in Math
- Understanding Knowledge Dynamics in LLM Explanations with a New Framework
Future Directions and Impact
This research introduces a powerful new framework for Knowledge Tracing, emphasizing the critical role of question content and the benefits of fine-tuning LLMs for this task. While NTKT currently requires more computational resources and was validated on a specific dataset (Eedi, mathematics, English), it opens up exciting avenues for future work. This includes adapting the framework to different subject areas, handling multi-modal content like diagrams, developing tools to interpret the LLM’s reasoning, and leveraging its generative capabilities to create personalized questions and explanations for students.
The findings suggest a promising future for AI in education, where models can understand student learning with unprecedented depth by truly comprehending the language of learning materials. For more details, you can refer to the full research paper: Next Token Knowledge Tracing: Exploiting Pretrained LLM Representations to Decode Student Behaviour.
