ST-DiffEye: Diffusion-based Continuous Gaze Generation via Joint Scanpath-Trajectory Modeling

2026-06-13

AI Summary

This work addresses the key challenge in eye movement modeling: effectively capturing the diversity of human gaze behavior in response to visual stimuli while jointly accounting for continuous trajectory dynamics and discrete scanpath structure. The study proposes, for the first time, a complementary representation that unifies gaze trajectories and scanpaths within a diffusion-based generative framework. By concatenating these representations as additional input channels, the method enables probabilistic generation of human-like fixations without modifying the backbone architecture. Furthermore, a distribution-aware evaluation framework based on Continuous Ranked Probability Score (CRPS) is introduced to better capture the intrinsic variability of gaze behavior. The approach achieves state-of-the-art performance on both task-driven visual search (with target-present and target-absent conditions) and free-viewing benchmarks, demonstrating the efficacy of joint modeling and distribution-aware assessment.

Abstract

We study the problem of human gaze modeling, which aims to generate the gaze patterns a viewer produces while observing a visual stimulus. Gaze is primarily captured through two modalities: continuous eye-tracking trajectories, which describe fine-grained motion dynamics, and discrete scanpaths, which describe high-level fixation structure. Because gaze varies substantially across viewers and trials, we treat this variability as a defining property rather than noise and model gaze as a stochastic generative process. Existing generative gaze models supervise on only one of these two representations in isolation. We hypothesize that trajectories and scanpaths describe gaze at complementary scales and are jointly informative during training, and test this hypothesis through ST-DiffEye, a joint trajectory-scanpath diffusion framework that couples both modalities by concatenating them as an additional raw input channel, requiring no architectural overhead beyond an input and output channel expansion. We further introduce a principled evaluation framework based on the Continuous Ranked Probability Score (CRPS), which generalizes any existing sequence similarity metric into a proper scoring rule that jointly assesses the accuracy and diversity of generated gaze. Experiments on task-driven visual search, covering both target-present and target-absent scenarios, and on free-viewing benchmarks demonstrate state-of-the-art performance. These results, along with detailed ablations, confirm the benefit of joint modeling and the value of distribution-aware evaluation in capturing the intrinsic variability of human gaze. Project webpage: https://st-diffeye.github.io/

Problem

• gaze modeling
• scanpath
• eye-tracking trajectory
• stochastic generative process
• human gaze variability

Innovation

• joint trajectory-scanpath modeling
• diffusion-based gaze generation
• stochastic generative process
• Continuous Ranked Probability Score (CRPS)
• distribution-aware evaluation