TLDR: This paper explores the design, deployment, and evaluation of an AI Instructor Agent as the primary teacher in a graduate-level Cloud Computing course at IISc. It details a pedagogical framework integrating the LLM-driven agent for content delivery, supported by human instructors for structure and Q&A. Early findings show students shift from broad topic exploration to deeper, focused inquiry, and an analytical framework evaluates engagement metrics like topic coverage, depth, and turn-level elaboration. The study highlights both the pedagogical benefits and practical challenges of using AI as a primary instructor in authentic classroom settings.
Artificial intelligence is rapidly transforming various sectors, and education is no exception. While many studies explore AI’s potential in learning, there’s a significant gap in understanding how these systems perform in real, active classroom environments, especially in higher education. A recent study, “Towards AI Agents for Course Instruction in Higher Education: Early Experiences from the Field,” by Yogesh Simmhan and Varad Kulkarni from the Indian Institute of Science (IISc), delves into this very challenge, offering early insights from deploying an AI Instructor Agent in a graduate-level Cloud Computing course.
The researchers designed and implemented an AI Instructor Agent, powered by a Large Language Model (LLM), to serve as the primary instructor for a cohort of 17 senior undergraduate, Masters, and PhD students. This innovative approach fundamentally shifts the traditional lecture-based teaching model to a student-driven, self-paced learning experience. The human instructor’s role evolved to designing the curriculum, providing initial context, and facilitating question-and-answer sessions, while teaching assistants handled hands-on lab sessions.
Integrating AI into the Classroom Workflow
The pedagogical framework developed for this study is highly structured and repeatable. The AI Instructor Agent, built within Microsoft Teams Copilot, is configured weekly with topic-specific instructions, while maintaining consistent course-level guidelines. This modular design ensures the agent remains focused on current topics while upholding a uniform pedagogical tone.
Each 90-minute in-person lecture session follows a distinct cycle: a human instructor introduces the week’s topics, setting the context and learning outcomes. Students then interact with the AI Instructor Agent on their laptops, exploring concepts, clarifying doubts, and taking non-graded quizzes at their own pace. Supplementary course slides are provided by the human instructor to offer visual aids and scope the breadth and depth of topics. After interacting with the agent, students engage in small group discussions to reinforce understanding and then participate in a human instructor-led Q&A session to address any remaining ambiguities or questions.
The AI agent’s configuration involves three key components: system-level prompts defining its persona and instructional goals, pedagogical alignment prompts guided by the Knowledge–Learning–Instruction (KLI) framework, and a topic knowledge base listing weekly topics with learning outcomes based on Bloom’s taxonomy. These ensure the agent is critical, accurate, and encourages reflective learning.
Measuring Student Engagement
A crucial aspect of this research is the Engagement Analytics Framework, which quantitatively evaluates student interactions with the AI Instructor. After each lecture, students submit their chat transcripts, which are then processed by a separate AI evaluation agent. This agent, configured with a structured system prompt, analyzes the transcripts consistently and unbiasedly, generating a report that summarizes engagement based on three primary metrics:
- Topic Coverage: The proportion of canonical subtopics a student engaged with, reflecting the breadth of their exploration.
- Topic Depth: How thoroughly a student explored each subtopic, rated on a four-level ordinal scale from ‘briefly mentioned’ to ‘examined in depth’.
- Turn Length: The number of words per student message, indicating elaboration and reflective effort.
These metrics are shared with individual students for personalized feedback and aggregated for the human instructor to provide insights into overall class engagement.
Early Findings and Observations
Preliminary results from the first two instructional modules (Virtualization and Container Runtimes, and Cloud Service and Deployment Models) revealed interesting trends. There was a noticeable contraction in topic coverage, decreasing from approximately 52.5% in Week 1 to 31.0% in Week 2. Concurrently, the average topic depth increased from 1.33 to 2.06. This suggests a shift in student behavior from broad conceptual exploration to more concentrated, deeper inquiry into selected topics.
The average turn length per topic also increased modestly, from 48.2 to 54.4 words, indicating that students’ responses became slightly longer and more elaborate over time. This aligns with the increase in topic depth, pointing towards more reflective and information-rich dialogues as students became more familiar with the conversational format.
Also Read:
- Enhancing Group Learning with AI: Introducing LLM Bazaar
- Evaluating AI Ethics Courses: A Multi-Perspective Approach with AI Assistance
Challenges and Future Directions
While promising, the study also highlighted several challenges. The researchers noted occasional hallucinations from the agent, such as providing non-existent URLs or incomplete diagrams. Students also expressed concerns about the optimal depth of coverage for each subtopic and how this self-paced learning would align with graded assessments. The reliance on a specific LLM model (Microsoft Copilot/GPT) also led to some students expressing a preference for alternatives like Google’s Gemini.
Despite these hurdles, the study demonstrated the potential for personalized learning, with one student successfully configuring the agent to provide responses in a “Bionic Reading” format. Future work will focus on correlating these engagement behaviors with actual learning outcomes, such as graded assessments, and integrating “Agentic AI” workflows for automated tool calling, fact verification, and adaptive feedback to enhance reliability and educational value. This research provides a foundational step towards a scalable, outcome-driven model for personalized AI-driven education. You can read the full paper here.
