TLDR: This paper introduces SAGA22, a new dataset for analyzing mathematics tutoring discourse using “talk moves.” It demonstrates that models initially trained on larger classroom teaching datasets can be effectively adapted and fine-tuned for tutoring settings, especially when incorporating longer dialogue context and speaker information. This approach significantly improves the automated analysis of tutor and student interactions, addressing the challenges of scaling up feedback for novice tutors.
Human tutoring plays a vital role in helping students learn, improve their academic performance, and foster personal growth. However, analyzing the rich conversations that happen during tutoring sessions, especially in mathematics, can be a complex and resource-intensive task. This is where the concept of “talk moves” comes into play – a framework for understanding different types of dialogue acts, rooted in the “Accountable Talk” theory.
The challenge lies in collecting, annotating, and analyzing large volumes of tutoring dialogues to develop effective machine learning models. Traditional methods often rely on skilled human observers, which are costly and time-consuming, making them inaccessible for widespread use, especially for training new tutors.
Bridging the Gap Between Classroom and Tutoring
Recent advancements in automated techniques have shown promise in detecting important features in educational discourse, such as productive dialogue and instructional talk. However, much of this earlier work has focused on traditional classroom settings, not the more intimate, small-group tutoring environment. This research paper addresses a crucial question: Can models originally designed for classroom teaching be adapted and used effectively for tutoring?
The paper specifically focuses on mathematics tutoring and the analysis of talk moves, which include both teacher/tutor and student dialogue acts. Research has consistently shown that the appropriate use of talk moves can significantly promote student learning and ensure equitable participation.
Introducing the SAGA22 Dataset
To tackle the differences between classroom and tutoring settings, the researchers developed a new mathematics tutoring dataset called SAGA22. This dataset comprises talk move annotations from 121 tutoring sessions. They then explored existing modeling strategies and datasets from classroom mathematics teaching to find the best ways to transfer this learning to their target tutoring domain.
SAGA22 was collected from high school tutoring sessions provided by Saga Education, a non-profit organization. These sessions operate on a hybrid model where students are in a physical classroom, but tutors work remotely, interacting through a virtual workspace equipped with video conferencing, speech, chat, and digital whiteboards. The dataset includes nearly 70 hours of video, with over 33,000 tutor utterances and 11,000 student utterances annotated with talk move labels.
The study also leveraged two previously published classroom teaching datasets: TALK MOVES and NCTE-119. While the distribution of talk moves in tutoring (SAGA22) is similar to classroom teaching, there are some differences. For instance, tutoring sessions tend to have more “None” labels (utterances not classified as a specific talk move) and fewer of other talk moves, possibly due to the higher proportion of high school recordings or less formal pedagogical training for tutors. Interestingly, student “Asking for More Information” (ASKMI) talk moves were more frequent in tutoring, suggesting closer interactions.
Modeling Strategies and Key Findings
The researchers employed a pretrain-finetuning approach, using the “RoBERTa-base” model as their foundation. They systematically investigated various modeling choices, including:
- Dialogue Context: Examining the impact of using only the previous utterance versus a longer context (previous 7 and subsequent 7 utterances).
- Speaker Information: Adding simple prefixes like “T:” for teacher/tutor and “S:” for student to indicate the speaker.
- Supplementary Pretraining Datasets: Exploring different combinations of the TALK MOVES, NCTE-119, and SAGA22 datasets for pretraining.
- Fine-tuning on SAGA22: Determining whether further fine-tuning on the SAGA22 dataset after pretraining improves performance.
The results were significant. The best tutor model achieved an 82.4 macro F1 score, comparable to existing models for classroom settings. This model was first pretrained on a combination of classroom-only datasets (TALK MOVES and NCTE-119) using longer context and speaker information, then fine-tuned on SAGA22. Similarly, the best student model achieved a 76.5 macro F1 score, outperforming existing baselines. This model was pretrained on all three datasets (including SAGA22) with longer context and speaker information, followed by fine-tuning on SAGA22.
Ablation studies revealed several important insights:
- Longer dialogue context generally improves performance, but it requires sufficient training data to be effective.
- Adding simple speaker information (e.g., “T:” or “S:”) consistently helps all models, especially with longer contexts.
- Supplementary pretraining on large teaching datasets significantly boosts performance, even allowing models to perform well on tutoring data without direct fine-tuning on the target dataset.
- Further fine-tuning on the smaller SAGA22 dataset generally helped tutor models but sometimes had mixed or even negative effects on student models, highlighting challenges like catastrophic forgetting.
The study demonstrates that even with a relatively small tutoring dataset, leveraging existing models and data from classroom teaching can lead to substantial improvements in automated talk move analysis. While the current models use a simplified speaker identification (all students as “S:”), the findings suggest that more fine-grained speaker modeling could further enhance personalized learning in tutoring settings.
Also Read:
- Decoding Classroom Dialogue: A Comprehensive Framework for Educational Feedback
- Enhancing Mathematical Reasoning in LLMs with Uncertainty-Driven Process Reward Models
Future Directions
The researchers acknowledge limitations, such as the datasets being English-only and limited to mathematics instruction. Future work will explore more comprehensive experiments with longer contexts, advanced large language models (LLMs), and better strategies for handling multi-party dialogue and fine-tuning to prevent forgetting. This research paves the way for more effective automated feedback tools for tutors, ultimately supporting better student learning outcomes. You can read the full research paper here: Enhancing Talk Moves Analysis in Mathematics Tutoring through Classroom Teaching Discourse.
