TLDR: This research paper explores the use of unimodal (BERT) and multimodal (AudiBERT) AI models for diagnosing Collaborative Problem Solving (CPS) indicators from dialogue. It finds that AudiBERT significantly improves CPS subskill classification, especially for sparse data, and that larger training datasets enhance recall for both models. The study proposes a human-AI complementarity framework where AudiBERT provides initial broad classifications, and BERT offers explainable, fine-grained indicator suggestions, emphasizing the importance of model explainability for human engagement in the coding process.
Detecting how students collaborate and solve problems, known as Collaborative Problem Solving (CPS), is a significant challenge in the field of AI in Education. Machine learning techniques are being explored to identify key indicators of CPS from student dialogues. Recent advancements have seen the use of Bidirectional Encoder Representations from Transformers (BERT) models on transcribed conversations to reliably detect these indicators.
A notable step forward was the introduction of AudiBERT, a multimodal variant of BERT that incorporates speech and acoustic-prosodic audio features to improve CPS diagnosis. While initial findings suggested improvements with multimodal data, the statistical significance of these enhancements wasn’t entirely clear, and there was a lack of guidance on how humans and AI could work together effectively in these diagnostic tasks.
A new workshop paper, titled Exploring Human–AI Complementarity in CPS Diagnosis Using Unimodal and Multimodal BERT Models, extends this prior research. It highlights that the AudiBERT model not only improved the classification of less common classes in the dataset but also showed statistically significant class-wise improvements over the standard BERT model, particularly for classifications within the social-cognitive dimension of CPS. However, similar significant improvements were not observed for classifications related to the affective dimension (emotions).
The study also conducted a correlation analysis, revealing that larger training datasets were strongly associated with higher recall performance for both AudiBERT and BERT models. This means both models were better at identifying all relevant instances when given more data to learn from. Interestingly, the precision of the BERT model (its ability to make correct positive classifications) was significantly linked to high inter-rater agreement among human coders, suggesting it performed better on tasks that humans found clearer and less complex to code.
When the BERT model was used to diagnose specific, fine-grained indicators within CPS subskills and affective states that were generally well-detected by AudiBERT, the performance was inconsistent across all indicators. This indicates that even within well-performing categories, some specific behaviors are harder for the AI to pinpoint.
A Structured Approach to Human-AI Complementarity
The paper concludes by outlining a structured approach to achieve human-AI complementarity for CPS diagnosis. The core idea is to actively engage human coders, using AI to help them achieve more consistent and reliable coding of CPS indicators. The proposed workflow suggests that human coders first perform initial coding on a subset of data to establish inter-rater reliability. Then, the AudiBERT model could provide an initial classification of the broader CPS subskill or affective state.
Following this, a human coder would be presented with the AI’s suggestion and asked to select a specific indicator. To guide this selection, a BERT model would provide a more fine-grained classification of the indicator, along with information on which tokenized words in the dialogue drove that classification. A warning would be issued if the suggested class was trained on a small dataset, prompting the human coder to consider its suitability. If the human coder is unsure, an alternative suggestion could be provided by the BERT model, especially for classes where human coders had high agreement (indicating less complexity).
The reason for using the BERT model for indicator-level classification, despite AudiBERT’s overall better performance, is the current lack of explainable AI (XAI) techniques for AudiBERT. Incorporating model explainability is crucial to support human agency and engagement in the reflective coding process, allowing researchers and practitioners to understand why the AI made a certain decision.
Also Read:
- Unpacking AI’s Decisions in Collaborative Learning: A Look Inside BERT’s Diagnostic Process
- Unraveling Complex Student Errors: A New Approach to Diagnosing Combined Steps in Tutoring Systems
Future Directions
The authors emphasize that this proposed human-AI complementarity scenario is currently hypothetical and requires rigorous empirical investigation and refinement. Future work includes designing an ensemble architecture that supports this approach and testing it with human coders. Additionally, developing an architecture that can explain the AudiBERT model’s classifications, including which audio features and tokenized words contribute to its accuracy, is a key area for future research. This would deepen the understanding of why multimodal models outperform unimodal ones and further support human-AI feedback loops in educational settings.
