LEXIC: Lightweight Eye-tracking eXtension via Injected Complexity

2026-07-09

AI Summary

Current purely eye-tracking–based models significantly underperform language model–based approaches in predicting reading comprehension performance. This work proposes a lightweight, language-model-free enhancement to the AhnCNN baseline on EyeBench by incorporating precomputed word-level difficulty signals—GPT-2 surprisal, word frequency, and word length—into fixation sequences through two novel mechanisms: direct concatenation (LEXIC-Concat) and residual bias modeling (LEXIC-Res). These mechanisms aim to capture both typical reader behavior and individual deviations. Evaluated on the OneStop dataset using seed ensembling and ten-fold cross-validation, the approach improves AUROC by 1.8–2.2 percentage points in the Unseen Text setting, with LEXIC-Concat yielding a statistically significant gain of 2.9 percentage points (p=0.010) in the Unseen Reader scenario.

Abstract

On the recent EyeBench benchmark, predicting reading comprehension from eye movements exposes a stark gap: text-aware models using pretrained language models reach 56--63% AUROC, while gaze-only models operate at chance. We ask how far a gaze-only model can be pushed by lightweight, language-model-free conditioning. Building on the EyeBench AhnCNN baseline, LEXIC-Base, we propose two mechanisms to inject three precomputed word-level difficulty signals, GPT-2 surprisal, word frequency, and word length, into the per-fixation input: direct concatenation, LEXIC-Concat, and a residual mechanism, LEXIC-Res, where a small head predicts typical-reader gaze response and the encoder is conditioned on the deviation. On the OneStop reading comprehension task, with K=5 seed-ensemble training across ten folds, both mechanisms produce statistically consistent AUROC gains on Unseen Text, +1.8 to +2.2 percentage points, Wilcoxon p <= 0.065. LEXIC-Concat additionally lifts Unseen Reader by +2.9 percentage points, p = 0.010. We trace an architectural boundary in LEXIC-Res on Unseen Reader, +1.8 percentage points, p = 0.19, to the prediction head being calibrated to training readers, transferring imperfectly to out-of-distribution readers.

Problem

• eye-tracking
• reading comprehension
• gaze-only models
• language-model-free
• EyeBench

Innovation

• eye-tracking
• reading comprehension
• language-model-free
• difficulty signals
• residual conditioning