TLDR: A study on SCRIPT, an AI chatbot for introductory programming, found that students follow a predictable sequence of feedback requests (task understanding, concepts, problem-solving steps, debugging). The chatbot successfully aligned with 75% of student requests, often providing additional helpful information. While designed to avoid full solutions, it sometimes provided them incrementally. The research offers insights into designing effective AI learning tools that balance guidance and flexibility.
A new study delves into how a chatbot named SCRIPT, designed to assist students in introductory programming courses, interacts with learners and whether it effectively meets their requests for feedback. Developed by Andreas Scholl and Natalie Kiesler from Nuremberg Tech, SCRIPT is built on ChatGPT-4o-mini and aims to support novice programmers by offering both open-ended conversations and structured guidance through predefined prompts.
The research involved 136 students from an introductory programming course at a major German university. Students used SCRIPT while working on programming tasks at home, and their chat interactions were automatically recorded. This allowed the researchers to analyze how students sought feedback and how the chatbot responded.
How SCRIPT Works
SCRIPT functions as a web application, integrating task descriptions and reference solutions directly into its system prompt, so students don’t need to manually copy and paste information. It offers two main ways for students to interact: “closed prompts” and “open prompts.” Closed prompts are predefined options based on established feedback types, guiding students through common problem-solving steps like understanding task constraints, identifying necessary programming concepts, recognizing mistakes, or determining how to proceed. Open prompts allow students to type in free-form questions, similar to a regular chatbot.
Crucially, SCRIPT is designed to prevent the generation of complete code solutions. Instead, it focuses on fostering step-by-step problem-solving by identifying errors without directly correcting them, providing hints, examples, or incomplete code templates. This approach aims to support learning rather than simply giving away answers.
Student Interaction Patterns
The study revealed a common sequence in how students requested feedback from SCRIPT. Students often started by asking about task constraints (KTC), followed by questions about necessary programming concepts (KC). As they progressed, they would inquire about how to proceed (KH), debug errors (KM), verify their results (KP, KR), and sometimes, at the end, ask for the correct solution (KCR). The chatbot typically responded with step-by-step explanations, often adding conceptual knowledge to guide students. Interestingly, when students used predefined closed prompts, SCRIPT gave more direct, binary feedback (e.g., correct/incorrect), which often led students to follow up with open prompts for more detailed explanations of their mistakes.
Matching Requests with Responses
A key finding was the high alignment between student requests and SCRIPT’s responses. In 47% of cases, the chatbot’s feedback directly matched what students asked for. When considering “over-matching” – where SCRIPT provided the requested feedback plus additional helpful information (like extra conceptual explanations or next-step guidance) – the alignment jumped to 75%. This indicates that SCRIPT generally understood and addressed students’ informational needs effectively. However, mismatches did occur in 22% of interactions, with a quarter of these happening when students requested direct solutions, which SCRIPT was programmed to deny.
Adherence to Design Principles
The research also assessed how well SCRIPT adhered to its design constraints. The chatbot largely succeeded in providing feedback one step at a time, though about a quarter of responses offered an overview with multiple steps but a clear starting point. While SCRIPT was designed not to give full solutions, it did provide them in 42 out of 100 requests for correct solutions, especially after a series of step-wise interactions. Simple code examples and structural templates were provided as intended, though templates sometimes became complete after multiple student inquiries. The chatbot also corrected student code directly in 34 instances, despite a constraint against it. Overall, SCRIPT’s responses were highly accurate, with 81% being fully correct and 14% partially correct.
Also Read:
- Unlocking Technical Interview Success with AI-Powered Think-Aloud Training
- Enhancing LLM Tutors for Long-Term Learning Outcomes
Implications for AI in Education
The study highlights the potential of GenAI-based chatbots like SCRIPT to support novice programmers. It suggests that a balance between structured guidance (closed prompts) and flexible interaction (open prompts) is beneficial. While the chatbot generally performed well, areas for improvement include tailoring conceptual explanations more precisely and refining performance feedback. The findings also underscore the importance of designing AI tools with pedagogical guardrails to encourage active learning rather than passive solution-seeking. For more details, you can read the full research paper available at this link.
