TLDR: AutoSIGHT is a novel system that uses eye-tracking data to automatically assess human expertise in visual tasks, demonstrated in iris presentation attack detection. It employs a multi-stream neural network to classify individuals as experts or non-experts in real-time, achieving an AUROC of 0.751 with just 5 seconds of observation and up to 0.831 with 30 seconds. This technology has broad implications for dynamic human-AI collaboration, adaptive learning, and various visual inspection applications.
In an increasingly AI-driven world, the ability to accurately assess human expertise, especially in visual tasks, is becoming vital for effective human-AI collaboration. A new research paper introduces AutoSIGHT, an Automatic System for Immediate Grading of Human experTise, designed to classify individuals as experts or non-experts based on their eye-tracking patterns while performing visual tasks.
The core idea behind AutoSIGHT is to teach machines to understand and evaluate human proficiency by analyzing how their eyes move and focus. This system builds upon an ensemble of features extracted from eye-tracking data, offering a novel approach to real-time expertise assessment.
How AutoSIGHT Works
The researchers, Byron Dowling, Jozef Porubcin, and Adam Czajka from the University of Notre Dame, developed AutoSIGHT using a multi-stream neural network classifier. This system processes several key eye-tracking metrics:
-
Average Fixation Duration (AFD): The average time a participant’s gaze remains on a single point.
-
Fixation Count (FC): The total number of times a participant’s gaze settles on a point.
-
Average Euclidean Distance (AED): The average distance between successive fixation points, indicating saccade movements and search behavior.
-
Raw Gaze Coordinates: The precise (x, y) locations of the participant’s gaze over time.
These features are fed into a sophisticated neural network that learns to differentiate between expert and non-expert visual behaviors. A sliding window approach is used, meaning the system continuously analyzes short segments of eye-tracking data (e.g., 5, 10, 15, 20, or 30 seconds) to provide a real-time assessment of expertise.
The Study: Iris Presentation Attack Detection
To test AutoSIGHT, the researchers applied it to the task of iris Presentation Attack Detection (PAD). This involves distinguishing between real iris images and fake ones (spoofs) that might be used to trick biometric systems. The study involved 6 expert participants (with backgrounds in iris recognition research or ophthalmology) and 53 non-expert participants (university students and staff).
Participants were shown various iris images and asked to verbally classify them as “Normal” or “Abnormal” while their eye movements were recorded using a PupilLabs PupilCore headset. This setup allowed for the collection of rich eye-tracking data, including raw gaze positions, fixation data, and derived metrics.
Also Read:
- Beyond Appearance: Verifying Identity in the Age of Photorealistic Avatars
- Unlocking Future Actions: INSIGHT’s Cognitive AI for Egocentric Video Understanding
Key Findings and Implications
The study yielded several significant insights:
-
Distinguishing Features: The research confirmed that Average Fixation Duration, Fixation Count, and Average Euclidean Distance are statistically significant indicators that differentiate experts from non-experts in visual tasks. Experts generally exhibited shorter average fixation durations, more fixations, and smaller overall Euclidean distances between fixations.
-
Real-Time Classification: AutoSIGHT successfully trained a classifier to distinguish experts and non-experts in real-time. Even with a small evaluation window of just 5 seconds, the system achieved an average Area Under the ROC curve (AUROC) performance of 0.751. When a larger evaluation window of 30 seconds was available, the AUROC increased to 0.831, demonstrating the model’s ability to leverage more information for improved accuracy.
-
Observation Time: While 30 seconds of observation provided the best overall performance, the results indicate that even shorter windows (like 5 seconds) are viable for applications requiring quicker decisions, with only a slight decrease in accuracy.
This work opens up new avenues for research, particularly in how human and machine expertise can be dynamically weighed in human-AI pairing setups. For instance, AutoSIGHT could enable AI systems to adapt their assistance based on a human’s real-time expertise level, or identify when an expert might be fatigued and needs replacement.
Potential applications extend beyond biometrics to areas like smarter source code review policies, where expert interactions could be monitored, or educational tools that dynamically adjust content difficulty based on a student’s real-time performance and engagement. The researchers have made their collected eye-tracking data, source codes, and trained models available to facilitate further research. You can find more details in the full research paper here.
AutoSIGHT represents a significant step towards creating more adaptive and trustworthy AI systems that can seamlessly collaborate with humans by understanding and responding to their dynamic levels of expertise.
