Eye on the Target: Eye Tracking Meets Rodent Tracking

2025-03-13

AI Summary

To address the low efficiency and high subjectivity of manual video annotation in animal behavior analysis, this paper proposes a novel paradigm for automated mouse instance segmentation that integrates eye-tracking with zero-shot segmentation. We pioneer the use of eye-gaze trajectories captured by Meta Aria smart glasses to generate initial prompt points, and design an intelligent prompt optimization algorithm incorporating geometric constraints and attention mechanisms—enabling high-precision segmentation without model fine-tuning. Our method eliminates reliance on manual annotations entirely. Evaluated on a mouse behavioral video dataset, it achieves a Jaccard index of 66.2%, representing a 70.6% relative improvement over baseline methods. The approach significantly enhances both segmentation accuracy and annotation efficiency, offering a scalable, low-barrier solution for video-based behavioral analysis.

Technology Category

• Computer Vision: Segmentation
• Machine Learning: Multi-instance/Multi-view Learning
• Humans and AI: Brain-Sensing and Analysis

Application Category

• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Semantics and Knowledge: Representation, semantic annotation, enhancement, enrichments, access and/or integration of a variety of data on the Web
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

Analyzing animal behavior from video recordings is crucial for scientific research, yet manual annotation remains labor-intensive and prone to subjectivity. Efficient segmentation methods are needed to automate this process while maintaining high accuracy. In this work, we propose a novel pipeline that utilizes eye-tracking data from Aria glasses to generate prompt points, which are then used to produce segmentation masks via a fast zero-shot segmentation model. Additionally, we apply post-processing to refine the prompts, leading to improved segmentation quality. Through our approach, we demonstrate that combining eye-tracking-based annotation with smart prompt refinement can enhance segmentation accuracy, achieving an improvement of 70.6% from 38.8 to 66.2 in the Jaccard Index for segmentation results in the rats dataset.

Problem

• Automate labor-intensive manual animal behavior annotation
• Improve segmentation accuracy using eye-tracking data
• Enhance segmentation quality with smart prompt refinement

Innovation

• Eye-tracking data generates prompt points
• Zero-shot segmentation model creates masks
• Post-processing refines prompts for accuracy