TLDR: A pilot study at George Mason University investigated how Generative AI (GAI) tools impact critical thinking in introductory Computational and Data Science courses. Researchers designed activities requiring students to evaluate and revise AI-generated content. The study found preliminary evidence that a video lesson on GAI output positively influenced critical thinking performance in one course (CDS 101), suggesting GAI can be integrated to foster deeper learning, despite concerns about over-reliance. The findings provide initial insights for more comprehensive future studies.
Generative Artificial Intelligence (GAI) tools have rapidly become a part of our daily lives, and their presence in higher education is no exception. While these powerful AI systems offer many benefits, their impact on a crucial skill—critical thinking—has remained a significant area for exploration. A recent pilot study conducted by researchers at George Mason University delves into this very topic, examining how students engage critically with AI-generated content in their coursework.
The study, titled “Pilot Study on Generative AI and Critical Thinking in Higher Education Classrooms,” was led by William Franz Lamberti, Samantha Rose Lawrence, Dominic White, Sunbin Kim, and Sharmin Abdullah. Their work addresses a gap in current research, moving beyond GAI as just a tutor or assignment completion tool to focus on how students evaluate the accuracy and appropriateness of AI outputs.
Given that GAI tools can sometimes produce information that is flawed or factually incorrect, the researchers designed specific learning activities. These activities challenged students in introductory Computational and Data Science courses to analyze, critique, and revise solutions provided by AI. The goal was to understand if structured critical thinking could be applied effectively when interacting with GAI.
Study Design and Participation
The research employed a quasi-experimental design, involving both treatment and control groups across two data science courses: CDS 101 and CDS 130. The “treatment” involved a 15-minute prerecorded video lecture designed to teach students how to evaluate GAI output, including concepts like “hallucinations” (when AI generates false information). The control group received the same homework assignment but without the video lesson.
Student participation was voluntary, and the study observed an overall participation rate of about 28% across all courses. This initial insight into student willingness to engage in such studies is valuable for future, larger-scale research.
Key Findings and Insights
While participation rates in the CDS 130 treatment classes were too low for a full statistical analysis, the researchers were able to perform an analysis on the CDS 101 sections. The results indicated that the video lesson had a statistically significant positive impact on student performance in the related GAI critical thinking assignment. This suggests that targeted instruction can indeed help students develop better critical thinking skills when interacting with AI.
However, the authors note that this evidence is considered “weak” due to the low sample size and other factors, such as assigning treatment and control in different sessions. Despite these limitations, the findings provide a valuable “weak prior” – an initial indication – for future, more comprehensive studies.
Also Read:
- The Effort Paradox: How Generative AI Can Deepen Student Learning
- Generative AI Prompts US College Students to Re-evaluate Higher Education’s Worth
Implications for Education
The pilot study underscores the potential for integrating GAI into the curriculum in a way that enhances, rather than diminishes, critical thinking. As GAI usage continues to rise among students for both academic and professional tasks, understanding how to guide them in critically assessing AI outputs becomes increasingly important. The findings lay the groundwork for educators to refine teaching strategies that foster deeper learning and academic integrity in an AI-driven world.
For more detailed information, you can read the full research paper here: Pilot Study on Generative AI and Critical Thinking in Higher Education Classrooms.
