Problem
Research questions and friction points this paper is trying to address.
state localization
object tracking
temporal grounding
image-text alignment
visual state retrieval
Déjà Cue: Localizing States in Object Histories via Vocabulary-Relative Coordinates
🤖 AI Summary
This work addresses the challenging problem of precisely localizing temporal intervals corresponding to specific visual states—such as empty/full or whole/cut—within an object’s state transition history. The authors propose a novel identity-conditioned moment retrieval task and introduce surrogate state descriptions to construct a lexical relative coordinate system, thereby reformulating state queries as object-specific relative measurement problems. This approach enables calibration of frame-level scores without additional training. By integrating a frozen encoder, state-balanced centroid correction, multi-scale temporal scanning, and an image–text similarity scoring mechanism, the method achieves substantial performance gains on 78 VOST sequences: R@1 (tIoU=0.5) improves from 10.3% to 20.5%, and Top-1 tIoU rises from 16.0% to 21.5%.
📝 Abstract
Tracking links observations of the same object through visual change, yet cannot by itself determine when the object is empty or filled, intact or cut. We formulate identity-conditioned state-moment retrieval: given a tracked-object history and alternative state descriptions, localize an interval in which each described state holds. Absolute image-text similarity scores descriptions independently; because every visible frame depicts the same target, shared object compatibility can obscure the state evidence needed to identify the target interval. The alternatives provide the missing reference: evidence for one state should be measured against the others. We introduce Déjà Cue, a training-free framework that turns these alternatives into a vocabulary-relative coordinate system. It subtracts their state-balanced centroid from each description, calibrates frame scores, and scans multiple durations within contiguous visible runs using a frozen encoder. On 78 VOST histories, holding the temporal scan fixed and changing only the query reference nearly doubles R@1 at tIoU 0.5 from 10.3\% to 20.5\% and raises Top-1 tIoU from 16.0\% to 21.5\%. Candidate-rank analyses show that vocabulary-relative queries rank useful intervals higher within the same candidate set. Related state descriptions can therefore serve as an object-specific, query-time coordinate system for reading frozen visual representations.
Innovation
Methods, ideas, or system contributions that make the work stand out.
vocabulary-relative coordinates
state-moment retrieval
training-free framework
frozen encoder
object history localization
🔎 Similar Papers
No similar papers found.
