TLDR: A new study explores using Visual Question Answering (VQA) AI models (LLaMA2, LLaMA3, QWEN3, NVILA) to monitor classroom activities from video. Researchers created a new dataset, BAV-Classroom-VQA, from real classroom recordings to evaluate these models. The results show promise for VQA in identifying behaviors and objects, especially for simple questions, but highlight challenges with complex reasoning. This technology could help teachers with semi-automated monitoring and improve teaching methods.
Classroom monitoring is essential for understanding student engagement and learning. Traditionally, this has relied on manual observation, which can be subjective and difficult to scale. However, recent advancements in Visual Question Answering (VQA) models are opening new doors for automatically analyzing complex classroom interactions from video recordings.
A new research paper, “Exploring the Application of Visual Question Answering (VQA) for Classroom Activity Monitoring,” delves into how these advanced AI models can be applied to educational settings. Authored by Sinh Trong Vu, Hieu Trung Pham, Dung Manh Nguyen, Hieu Minh Hoang, Nhu Hoang Le, Thu Ha Pham, and Tai Tan Mai, the study investigates the potential of several leading open-source VQA models, including LLaMA2, LLaMA3, QWEN3, and NVILA, in the context of analyzing student behavior in classrooms.
A New Dataset for Real-World Classrooms
To rigorously test these models, the researchers introduced a new dataset called BAV-Classroom-VQA. This unique dataset was created from actual classroom video recordings at the Banking Academy of Vietnam. The paper details the process of collecting and annotating this data, which includes diverse question-answer pairs reflecting real educational scenarios. This dataset is crucial because existing VQA datasets don’t specifically focus on the complex dynamics of classroom environments.
How VQA Models Perform in the Classroom
The study benchmarked the performance of the selected VQA models on this new dataset. The initial results are promising, showing that all four models can effectively answer behavior-related visual questions. This demonstrates their potential for future classroom analytics and intervention systems.
For “Close-Ended” questions, which have predefined answers like “Yes/No” or multiple-choice, models like LLaMA2 and QWEN3 showed strong accuracy, especially for simple binary tasks. QWEN3 also stood out for its fast response times, making it suitable for quick, direct answers. However, “Single Choice” questions proved more challenging for all models.
When it came to “Open-Ended” questions, which require more detailed descriptions or explanations, the models faced greater complexity. While they performed well on factual questions (e.g., identifying objects) and object detection, they struggled more with tasks requiring spatial reasoning, counting, action recognition, and higher-level inference. NVILA and QWEN3 generally achieved higher scores for these types of questions, indicating their strength in semantic understanding and structural matching.
Also Read:
- MoHoBench: A New Benchmark for Evaluating Honesty in Multimodal AI
- YourBench4Edu: Tailoring Reading Comprehension Questions for Young Learners
The Future of Classroom Monitoring
The research concludes that VQA systems hold significant practical potential for assisting teachers with semi-automated classroom monitoring. They can offer real-time insights into student behaviors, attention levels, and interactions during lectures, ultimately contributing to improved teaching and learning management. While current models show limitations in complex reasoning tasks, their ability to handle quick recognition tasks like counting students or detecting object usage is a significant step forward.
Looking ahead, the researchers propose several directions for future work, including expanding the dataset with more diverse classroom videos, improving models through domain-specific fine-tuning, exploring newer multimodal models like GPT-4o, and developing real-time VQA systems. This research paves the way for a more objective and scalable approach to understanding and enhancing the educational experience. You can read the full research paper here.
