Whareformer: Learning to Track What is Where in Long Egocentric Videos

2026-07-09

AI Summary

This work addresses the “Out of Sight, Not out of Mind” (OSNOM) task—online 3D tracking of objects in egocentric videos even after they exit the field of view or become heavily occluded—and proposes Whareformer, the first end-to-end Transformer model tailored for this challenge. Whareformer jointly models object appearance (“what”) and dynamic 3D location (“where”) through an updatable memory module, and efficiently handles newly emerging objects via a feedforward trajectory assignment mechanism and dedicated trajectory tokens. Leveraging relative distance cues and evolving trajectory representations, the model generalizes effectively to long videos across multiple datasets despite training on a minimal dataset. Evaluated on 260 test videos from EPIC-KITCHENS-100, IT3DEgo, and HD-EPIC, Whareformer significantly outperforms existing approaches, achieving state-of-the-art performance.

Abstract

The recently established 'Out of Sight, Not out of Mind' (OSNOM) task for egocentric videos focuses on tracking objects that are moved by the camera wearer, online, maintaining knowledge of instance locations throughout the video even when they leave the field of view or become heavily occluded. In this paper, we propose the first learning-based solution to the OSNOM task: Whareformer, a transformer-based model with two components: an updatable memory of established tracks and a track assignment module that associates observations with existing tracks in a feed-forward manner. Whareformer jointly reasons over evolving object appearance (what) and updated 3D location (where), and employs a dedicated New Track token to reason about novel objects. Thanks to its design choices of using relative distances and evolving track representations, Whareformer is trained on a small set of 56 videos but achieves SOTA performance on 260 long test videos from three datasets: EPIC-KITCHENS-100 (unseen videos), IT3DEgo, and HD-EPIC, with significant absolute improvements over prior work.

Problem

• egocentric videos
• object tracking
• out of sight
• 3D location
• occlusion

Innovation

• Whareformer
• OSNOM
• transformer-based tracking
• egocentric video
• 3D object localization