TLDR: A new study explores the application of ChatGPT 3.5 for generating quiz questions in higher education, specifically for a ‘Corporate Finance I’ course. Researchers designed specific prompting patterns for multiple-choice, true-false, and calculative questions. A ‘Blind Test’ survey involving lecturers and students revealed that while lecturers could largely distinguish AI-generated questions, multiple-choice questions created by ChatGPT were often mistaken for human-made ones. The study highlights ChatGPT’s potential to streamline question bank creation, despite some limitations in generating complex, logically linked questions.
Large language models, or LLMs, are rapidly changing many aspects of our daily lives, including how businesses operate and how we interact with technology. In the field of education, these powerful AI tools are beginning to show significant potential, especially in automating tasks that are often time-consuming for educators.
A recent research paper, titled “A ChatGPT-Based Approach for Questions Generation in Higher Education,” explores how ChatGPT, a popular LLM, can be used to help university educators create quiz questions and assess students. The study was conducted by a team of researchers: Sinh Trong Vu, Huong Thu Truong, Oanh Tien Do, and Tu Anh Le from the Banking Academy of Vietnam, along with Tai Tan Mai from Dublin City University. Their work focuses on developing effective ways to prompt ChatGPT to generate a diverse range of questions, aiming to streamline the process of building question banks for higher education.
The researchers chose ChatGPT 3.5 for their study for several practical reasons. It is freely accessible, which helps keep research costs down. Furthermore, ChatGPT 3.5 has a proven track record of generating high-quality, human-like text and is widely used, making it a reliable choice for this kind of application.
To test ChatGPT’s capabilities, the team focused on the subject “Corporate Finance I,” a compulsory course at the Banking Academy of Vietnam. This subject was ideal because it involves various question types, including Multiple Choice Questions (MCQs), True-False statements, and Real-Scenario/Calculative exercises. The researchers meticulously designed specific prompting patterns for each question type, guiding ChatGPT to produce relevant and structured questions. For instance, they used a combination of elements like defining ChatGPT’s ‘Role’ (e.g., ‘You are a lecturer’), specifying the ‘Task’ (e.g., ‘Please create questions’), providing ‘Context’ (e.g., ‘focus on the content of [lesson name]’), giving an ‘Example’ of the desired question, and defining the ‘Format’ for the output.
After generating a large set of questions, the researchers performed a preliminary self-assessment. They found that some questions were duplicated, and a notable percentage of calculative exercises lacked sufficient data to be solvable. This highlights a current limitation of LLMs, which sometimes rely on word probability rather than logical completeness. From a total of 390 generated questions, 74 were removed due to these issues.
To objectively evaluate the quality of the AI-generated questions, the team conducted a “Blind Test.” This involved mixing questions created by ChatGPT 3.5 with questions from the Banking Academy’s official “Corporate Finance I” workbook. A survey form with 15 mixed questions was then given to 91 participants, including lecturers, students who had completed the course, students currently studying it, and students who had not yet started the subject. Participants were asked to identify whether each question was created by a human or by ChatGPT.
The results of the “Blind Test” were quite insightful. Lecturers showed a high ability to distinguish between human-generated and AI-generated questions, correctly identifying them in about 13.5 out of 15 cases. Students who had completed the course performed better at distinguishing than those currently studying, and both groups were better than students who had not yet started the subject. This suggests that familiarity with the course content helps in recognizing the nuances of human-created questions.
Interestingly, the study found that multiple-choice questions generated by ChatGPT were the hardest for participants to correctly identify as AI-generated, with only about 30% accuracy. This indicates that ChatGPT is particularly effective at creating concise MCQs that closely resemble those made by humans. The paper suggests that AI models can generate plausible incorrect answers (distractors), making MCQs more challenging to differentiate. However, True-False and Real-Scenario/Calculative questions were more easily identified as AI-generated, often lacking structured links to other concepts or overall reasoning, and sometimes being vague or having minor language inaccuracies in the Vietnamese version.
Also Read:
- AI’s Role in College Math: A Comparative Study of Human and Artificial Intelligence Capabilities
- AI Literacy Shapes Student Experience with AI-Powered Assessments, Study Finds
In conclusion, this research demonstrates the significant potential of ChatGPT 3.5 in assisting educators with question generation for student assessment in higher education. While there are still some limitations, particularly with complex question types requiring deep conceptual links or precise data, the ability of LLMs to create high-quality questions, especially MCQs, is promising. The authors hope to integrate their methodology with other scientific solutions in the future to further enhance the efficiency and quality of question bank creation, ultimately saving time and effort for lecturers. You can read the full research paper here: A ChatGPT-Based Approach for Questions Generation in Higher Education.
