VistaRef: Boosting Visual Spatial Orientation Awareness for Pointing-to-Object Detection

2026-06-23

AI Summary

Existing Transformer-based vision models struggle to accurately model the implicit pointing ray encoded in finger poses, leading to localization drift and ambiguity in scenarios involving distant or densely packed objects. To address this, this work proposes a Local Hand Entity Modeling (LHEM) module and a Geometric Ray Modeling (GRM) module, along with a novel Orientation Consistency Alignment Loss (OCAL), thereby explicitly integrating multi-view geometric priors into the pointing detection task for the first time. By combining hand-pose embeddings, ray-aware feature modeling, and attention-guided feature fusion within the Transformer architecture, the proposed method significantly enhances spatial localization accuracy, achieving an absolute improvement of 14 percentage points in pointing target detection performance.

Abstract

Grounding deictic gestures in natural images is fundamental to AR and human-robot collaboration, providing a basis for seamless spatial interaction. While Transformer-based visual models have achieved significant progress in general object detection, their global attention mechanisms often neglect micro-geometric relationships, degrading orientation accuracy. In pointing tasks, this deficiency manifests as an inability to accurately capture the pointing ray implied by finger poses, which results in pointing drift and localization ambiguity when dealing with distant or densely packed objects. To address this, we propose VistaRef, a framework designed to explicitly enhance spatial orientation awareness. First, we develop the Local Hand Entity Modeling (LHEM) module, which incorporates hand-pose embeddings to strengthen the model's capability to capture subtle finger deviations. Second, drawing inspiration from multi-view geometry, we construct the Geometric Ray Modeling (GRM) module to transform implicit orientation information into explicit spatial geometric features, guiding feature aggregation and deep fusion via attention mechanisms. Furthermore, we introduce a novel Orientation-Consistent Alignment Loss (OCAL) to synergistically supervise hand presence and pointing consistency, ensuring that all architectural improvements collectively serve the core objective of spatial localization. Experimental results demonstrate that VistaRef significantly outperforms the baseline, achieving a 14-point absolute gain in grounding accuracy. Qualitative analysis further confirms that VistaRef effectively models the geometric correlation from hand to target, bridging the spatial perception gap inherent in traditional Transformers for complex scenarios. Code: https://github.com/lingli1724/VistaRef.

Problem

• pointing-to-object detection
• spatial orientation awareness
• deictic gesture grounding
• pointing drift
• localization ambiguity

Innovation

• spatial orientation awareness
• hand pose embedding
• geometric ray modeling
• deictic gesture grounding
• orientation-consistent alignment