TLDR: A study investigated how 34 students interacted with an AI-powered next-step hint system in an IDE for Kotlin programming. It identified 16 common interaction patterns, noting that 40% of interactions were ‘neutral’ and many ‘negative’ ones were due to system errors. Interviews with 6 students revealed two key strategies for handling unhelpful hints: ‘Selective Use of Hints,’ where students manually applied only parts of suggestions, and ‘Combining Partial Solutions,’ where they generated multiple hints by modifying their code and integrated insights from various suggestions. The findings offer insights for improving AI hint design in education.
Automated feedback is becoming an increasingly vital component in modern computer science education, offering personalized learning experiences. Among the various forms of feedback, next-step hints are particularly valuable as they guide students with actionable steps to solve programming tasks. A recent study delves into how students engage with an AI-powered next-step hint system within an integrated development environment (IDE) for learning.
The research, titled “Understanding Student Interaction with AI-Powered Next-Step Hints: Strategies and Challenges,” was conducted by Anastasiia Birillo, Aleksei Rostovskii, Yaroslav Golubev, and Hieke Keuning. Their work sheds light on the intricate ways students interact with these intelligent systems, revealing both common behavioral patterns and ingenious strategies for overcoming unhelpful guidance.
Exploring Student Behavior with AI Hints
The study aimed to answer two key questions: what behavioral patterns emerge from student interactions with the hint system, and what strategies do students use when hints aren’t helpful? To investigate this, the researchers collected extensive data from 34 first- and second-year Bachelor’s students as they tackled Kotlin programming tasks. This rich dataset included detailed logs of keystrokes, IDE actions, and every interaction with the hint system, totaling millions of events. Additionally, semi-structured interviews were conducted with 6 students to gain deeper qualitative insights.
The AI-driven hint system, integrated into the JetBrains Academy plugin, offers two levels of assistance: textual hints and code hints. When a student encounters a problem, they can request a textual hint outlining a single actionable step. If more help is needed, a code hint provides the exact code fragment. This system uniquely combines large language models (LLMs) with static analysis to ensure hints are granular and high-quality.
Interaction Patterns and Challenges
Using process mining techniques, the researchers identified 16 common interaction scenarios. They categorized hint-request sessions into positive (hint accepted), neutral (text hint viewed, no explicit acceptance or rejection), and negative (hint declined or system error). The analysis showed that 28.48% of interactions were positive, 40% were neutral, and 27.05% were negative. Interestingly, a significant portion of negative interactions stemmed from system errors, such as internet connection issues or LLM provider problems, rather than the hint’s quality itself.
A notable finding was that students often continued to use the system even after encountering errors, sometimes returning after a significant break. This suggests a persistence in seeking help despite technical glitches. The study also observed instances where students repeatedly clicked the ‘Show Code Hint’ button, possibly indicating confusion about its functionality or location within the IDE.
Also Read:
- Bridging the AI Literacy Gap: How Community College Instructors Envision Teaching AI to All Students
- Experiential AI Learning: Instructors Share Insights on Engaging Non-STEM Students with Real-World Scenarios
Strategies for Overcoming Unhelpful Hints
The interviews provided crucial insights into how students manage hints that aren’t immediately useful. Two primary strategies emerged:
- Selective Use of Hints: Some students chose not to blindly accept code hints. Instead, they would manually apply only a portion of the suggested code, adapting it to better fit their understanding or to refine their solution. For example, a hint might suggest trimming an image and storing the result in a new variable, but a student might only implement the trimming part, finding it sufficient. This demonstrates that students actively analyze and integrate hints rather than just copying them.
- Combining Partial Solutions: Other students developed a strategy of making small modifications to their code to generate multiple hints. They would then observe these different suggestions, often keeping previous code hint windows open for reference, and combine insights from several hints to construct a working solution. This iterative process of tweaking code and requesting new hints allowed them to troubleshoot and progress effectively, even when individual hints were insufficient.
These findings highlight that students are resourceful and adaptive in their learning process, often using the hint system in ways not explicitly designed. The study suggests that future systems could benefit from offering multiple hints with different approaches simultaneously, empowering students to construct solutions more effectively.
The research provides valuable insights into student behavior with AI-powered next-step hints, offering a foundation for improving the design of such systems to better support learning in computer science education. The detailed dataset collected during this study has been made publicly available to foster further research in this area. You can find the full paper here: Understanding Student Interaction with AI-Powered Next-Step Hints: Strategies and Challenges.
