Gold Points Sniper: Self-guided Visual Reasoning in VLM for Fine-grained Action Understanding

2026-06-21

AI Summary

Existing approaches struggle with fine-grained action understanding in wide-angle scenes: open-vocabulary recognition is constrained by predefined labels, while vision-language models (VLMs) often fail to balance informational richness with factual accuracy. To address this, this work proposes the Gold Points Sniper framework, which uniquely integrates self-guided reasoning with semantic entailment evaluation. The framework establishes an end-to-end lightweight multimodal reasoning pipeline comprising three modules—keypoint extraction, selective questioning, and factual consistency assessment. This approach substantially enhances fine-grained action comprehension for lightweight VLMs on the CAP benchmark, generating information-dense descriptions while preserving higher factual fidelity, with certain model variants approaching the performance of GPT-4o.

Abstract

Robots operating in everyday environments must understand fine-grained human actions, intentions, and contextual cues from broad views where people occupy only small regions, a capability unmet by current systems. While open-vocabulary action recognition methods remain limited to assigning predefined labels, and vision-language models (VLMs) face an inherent trade-off between informational richness and factual fidelity in their outputs, neither approach achieves the deep semantic interpretation required for reliable human-robot interaction. We propose Gold Points Sniper (GPS), a novel framework that empowers lightweight VLMs with self-guided multimodal reasoning capabilities for fine-grained human action understanding. Our approach comprises three key modules: Gold Points Extractor trains VLMs to identify critical action-relevant details, Selective Socratic Questioner validates and refines these details through selective self-questioning, and Semantic Entailment Evaluator quantitatively assesses factual consistency using semantic entailment classification. Extensive experiments on our curated instruction-tuning dataset based on the CAP benchmark demonstrate that GPS-enhanced lightweight VLMs achieve substantial performance improvements, with some models reaching performance comparable to proprietary GPT-4o while maintaining superior factual accuracy. Our work establishes a reliable foundation for fine-grained action understanding in domestic robotics, enabling robots to safely interpret human behavior through information-dense yet factually grounded descriptions. Source code, training configurations, annotation prompts, and dataset details are released at https://github.com/Haodi-Liu/GPS-Gold-Point-Sniper.

Problem

• fine-grained action understanding
• vision-language models
• human-robot interaction
• factual fidelity
• open-vocabulary action recognition

Innovation

• self-guided reasoning
• fine-grained action understanding
• vision-language models
• semantic entailment
• Gold Points Sniper