AutoSIGHT: Automatic Eye Tracking-based System for Immediate Grading of Human experTise

2025-08-01

AI Summary

This study addresses the challenge of dynamically allocating human–machine roles in collaborative systems by proposing the first framework for automatic, real-time assessment of human expertise in visual tasks using ultra-short-duration eye movement data. The method discriminates experts from non-experts using only 5 seconds of gaze trajectories and progressively improves classification performance as the temporal window extends. By extracting multidimensional oculomotor features and integrating them via ensemble learning models—evaluated using the Area Under the Receiver Operating Characteristic curve (AUROC)—the framework achieves AUROC scores of 0.751 (5 s) and 0.831 (30 s) on the iris presentation attack detection task, significantly outperforming baseline approaches. Its core contribution is the first demonstration of real-time, quantitative expertise assessment from sub-5-second eye-tracking data, establishing a deployable, capability-aware paradigm for human–machine collaboration systems.

Technology Category

• Computer Vision: Motion & Tracking
• Humans and AI: Human-in-the-loop Machine Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

Can we teach machines to assess the expertise of humans solving visual tasks automatically based on eye tracking features? This paper proposes AutoSIGHT, Automatic System for Immediate Grading of Human experTise, that classifies expert and non-expert performers, and builds upon an ensemble of features extracted from eye tracking data while the performers were solving a visual task. Results on the task of iris Presentation Attack Detection (PAD) used for this study show that with a small evaluation window of just 5 seconds, AutoSIGHT achieves an average average Area Under the ROC curve performance of 0.751 in subject-disjoint train-test regime, indicating that such detection is viable. Furthermore, when a larger evaluation window of up to 30 seconds is available, the Area Under the ROC curve (AUROC) increases to 0.8306, indicating the model is effectively leveraging more information at a cost of slightly delayed decisions. This work opens new areas of research on how to incorporate the automatic weighing of human and machine expertise into human-AI pairing setups, which need to react dynamically to nonstationary expertise distribution between the human and AI players (e.g. when the experts need to be replaced, or the task at hand changes rapidly). Along with this paper, we offer the eye tracking data used in this study collected from 6 experts and 53 non-experts solving iris PAD visual task.

Problem

• Automatically classify human expertise using eye tracking data
• Assess iris Presentation Attack Detection performance via eye movements
• Improve human-AI pairing by dynamically weighing expertise levels

Innovation

• Eye tracking-based expertise classification system
• Ensemble features from eye tracking data
• Dynamic human-AI expertise weighing