TLDR: A new study introduces a ten-item, indirect-elicitation instrument to measure ‘policy-practice voids’ in academia regarding generative AI. It identifies three key indicators: AI-integrated assessment capacity, sector-level necessity, and ontological stance, providing institutions with auditable signals to inform assessment, curriculum, and procurement decisions. The framework helps understand the gap between institutional rules and how academics actually use AI, offering a reusable measurement layer.
The rapid spread of generative AI in academic settings is creating a significant challenge: a growing gap between institutional policies and how academics actually use these powerful new tools. This divergence, termed ‘policy-practice voids,’ can have serious consequences, especially concerning assessment integrity and the future of education. A new study introduces a novel approach to measure these voids, offering a compact, auditable instrument designed to provide clear indicators for institutions.
Traditionally, digital tools in academia, like search engines or reference managers, were peripheral and easily compartmentalized. Generative AI, however, targets core academic activities such as language, computation, and analysis. It’s readily available off-platform and embedded in everyday software, making bans difficult to enforce and usage hard to police. This ubiquity highlights the need for a dedicated tool to understand and measure the disconnect between what policies dictate and what practitioners do.
The research, led by R. Yamamoto Ravenor from Tokyo Women’s Medical University, proposes a ten-item, indirect-elicitation instrument embedded within a structured interpretive framework. This framework is designed to extract both empirical (practice-focused) and epistemic (concept-focused) signals from academics. Unlike direct surveys that can be influenced by social desirability, this instrument uses indirect questions to reveal meaningful behavioral patterns.
Also Read:
- Cultivating Human Intelligence in the Age of AI: A New Educational Paradigm
- AI Helps Instructors Anticipate Student Struggles in Challenging Computer Science Courses
Three Key Indicators for Institutions
The study distills these signals into three filtered indicators, each offering unique insights into the policy-practice void:
1. AI-integrated assessment capacity: This indicator identifies academics who are skilled in using AI, perceive its benefit in teaching, and are confident in detecting AI-generated content. Among this group, the study measures the proportion who would fully allow AI use in exams. A high prevalence here suggests that institutions might have untapped capacity for AI-resilient assessment that is currently unused due to a lack of clear authorization.
2. Sector-level necessity: This indicator focuses on rigorous AI users – those who tightly control AI outputs (editing, rewriting) but still credit AI with a high contribution to their work. Within this group, the study gauges the proportion who believe AI could challenge the relevance of established disciplines. This signal suggests that for some academics, AI is more than just a useful tool; it’s seen as a transformative force. Such necessity signals might be overlooked or discounted in environments where AI adoption is still contested.
3. Ontological stance: This indicator delves into academics’ fundamental views on AI. It focuses on respondents who see AI as different in kind from prior tools and report changes in their academic practice due to AI. Among these, it distinguishes between a ‘material’ view of intellect (AI differs from human intellect in degree) and an ‘immaterial’ view (AI differs in kind). This distinction is crucial for procurement decisions. For instance, a material view might support claims for AI based on quantitative gains (e.g., throughput), while an immaterial view could support claims related to qualitative aspects like judgment or disciplinary formation, requiring different evidentiary standards. Without clear rubrics, procurement approvals risk becoming arbitrary.
A pilot deployment with 214 academics illustrated how this instrument yields structured, interpretable indicators. For example, the pilot found that 55% of academics demonstrating AI skill, perceived teaching benefit, and detection confidence would fully allow AI in exams. Furthermore, among high-control users who credited AI with high contribution, 81% believed AI could challenge established disciplines. These findings underscore the practical utility of the instrument for institutional planning.
The study emphasizes that its aim is methodological: to prototype a minimal, auditable measurement layer. While the pilot results are illustrative and not intended for population inference, they demonstrate the instrument’s ability to reveal practitioner-side capacities and dispositions with clear implications for assessment, curriculum design, and procurement strategies. By providing transparent, reproducible indicators, institutions can better monitor trends, support staff development, and make informed decisions as generative AI continues to reshape higher education.
For more detailed information on the methodology and findings, you can access the full research paper: Academics and Generative AI: Empirical and Epistemic Indicators of Policy–Practice Voids.
