EgoHieraLoc: A Cortically Inspired Hierarchical Segmentation-Guided Framework for Egocentric Visual Query Localization

2026-08-10

AI Summary

This work addresses the challenge of visual query localization in first-person videos, where ambiguous object boundaries and insufficient global contextual guidance hinder performance. Inspired by the hierarchical perceptual mechanisms of the human cortex, the authors propose a unified 2D/3D visual query localization framework. The method leverages segmentation priors to extract foreground-aware query representations and employs deformable correlation filters for robust localization, further refining boundaries through multi-scale region-adaptive contextual feedback. In 3D scenes, it innovatively introduces a geometry–semantics joint confidence measure to evaluate and fuse multi-view information. To the best of our knowledge, this is the first study to integrate hierarchical perception with geometry–semantics confidence modeling for this task, achieving state-of-the-art performance on both VQL-2D and VQL-3D benchmarks.

Abstract

Visual query localization (VQL) aims to retrieve and re-localize a queried object in egocentric videos, yet remains challenging when object boundaries are ambiguous and global context cannot effectively guide fine-grained localization. Human vision handles such ambiguity through a hierarchical process: it rapidly screens foreground candidates, selectively attends to the target despite distractors, refines perception via feedback between global context and local detail, and, when a single view is unreliable, integrates evidence across viewpoints according to its credibility. Inspired by these competencies, we propose \textbf{EgoHieraLoc}, a unified framework for VQL-2D and VQL-3D. A Discriminative Parsing Module first extracts foreground-aware query representations using segmentation priors; a Query-Aware Module then performs robust target localization through discriminative correlation filtering with deformable modeling; and a Regional Adaptation Module feeds multi-scale context back into local regions to recover precise object boundaries. To extend this perceptual hierarchy to 3D localization, we introduce Geometric-Semantic Joint Confidence (GSJC), which multiplicatively couples segmentation confidence with local depth consistency, multi-view back-projection consistency, and triangulation-baseline quality, so that a viewpoint contributes to the 3D estimate only when it is credible both semantically and geometrically. Extensive experiments demonstrate state-of-the-art performance on both VQL-2D and -3D benchmarks.

Problem

• egocentric vision
• visual query localization
• ambiguous object boundaries
• fine-grained localization
• multi-view consistency

Innovation

• hierarchical segmentation
• discriminative correlation filtering
• geometric-semantic joint confidence
• egocentric visual query localization
• multi-view consistency