TLDR: A new research paper introduces custom Large Language Models (LLMs) that can measure teaching quality with human-level and even super-human performance. By using sentence-level embeddings, these specialized LLMs can reliably apply complex classroom observation instruments to long-form transcripts. The models’ aggregate scores align with teacher value-added measures, indicating they capture features relevant to student learning. This methodology offers a scalable and reliable path for AI-driven instructional measurement, though further validation is needed for broader application.
Measuring the quality of teaching has long been a complex and challenging endeavor. Traditional methods, often relying on expert human observers, suffer from issues like low reliability, inconsistent accuracy, and high costs. While Large Language Models (LLMs) have shown promise in various fields, general-purpose models have struggled to effectively apply the intricate and nuanced classroom observation instruments needed for accurate teaching assessments.
A recent research paper, Measuring Teaching with LLMs, introduces a novel approach to overcome these limitations. Authored by Michael Hardy from Stanford University, this study proposes using custom LLMs built on sentence-level embeddings. This architecture is particularly well-suited for analyzing long-form classroom transcripts, which often involve complex, interpretive dialogue, unlike the more granular subword tokenization used by conventional LLMs.
The researchers systematically evaluated five different sentence embeddings under a training regimen designed for data efficiency and to prevent overfitting. The results are significant: these specialized models achieved performance levels comparable to, and in some cases even surpassing, expert human ratings. Specifically, they demonstrated correlations with expert human ratings above 0.65, exceeding the average correlation observed between human raters themselves.
A key insight from the study involves how these advanced models interpret teaching quality. Through an analysis of annotation context windows, the researchers found that models more aligned with human judgments attributed a larger portion of score variation to broader, lesson-level features rather than isolated statements or utterances. This finding challenges the common practice of single-turn annotation paradigms, suggesting that a more holistic, temporally-aware approach is necessary for accurate assessment.
To ensure the practical relevance and external validity of their findings, the study also examined how aggregate model scores aligned with teacher value-added measures (VAMs). VAMs are considered a gold standard for estimating a teacher’s impact on student achievement gains. The research found that the aggregate model scores did align with these student learning outcomes, indicating that the models are indeed capturing features relevant to student learning. However, this alignment did not hold true at the individual item level, suggesting that while the models learn useful signals, they have not yet achieved full generalization across all specific instructional dimensions.
This work establishes a powerful new methodology for AI-driven instructional measurement. It offers a promising path toward providing scalable, reliable, and valid feedback for educator development, potentially transforming how teaching quality is assessed and improved. The paper’s contributions include a longitudinal analysis of how LLMs learn to understand effective instruction, a new benchmark for automated instructional rating, a critique of single-turn evaluation methods, and a novel framework linking LLM ratings to student achievement.
The study utilized data from the National Center for Teacher Effectiveness (NCTE) Main Study, which includes transcripts and expert ratings of math instruction from fourth and fifth-grade classrooms. Two observation frameworks were used: the CLASS framework for general instructional practices and the Mathematical Quality of Instruction (MQI) for mathematics-specific teaching. The custom encoder models were designed with shared weights across 25 distinct teaching dimensions, allowing them to learn both general and specific features of instructional quality.
Also Read:
- Guiding AI Math Skills with Self-Optimizing Thought Vectors
- Unveiling LLM’s Inner Workings: How Fractal Geometry Measures Text Complexity
While the findings are highly encouraging, the authors acknowledge several limitations. The dataset was specific to fourth and fifth-grade mathematics classrooms in the United States, meaning the models’ generalizability to other grade levels, subjects, or international contexts is yet to be proven. The approach was also tailored to a subset of rating items from the MQI rubric, and the models are custom-built encoders, not general-purpose GPT-style LLMs. Therefore, further research and validation are necessary before these methods can be considered for high-stakes, real-world implementation.
