IntentVLM: Open-Vocabulary Intention Recognition through Forward-Inverse Modeling with Video-Language Models

2026-04-26

AI Summary

Social robots struggle to robustly infer human intentions in multimodal settings. To address this challenge, this work proposes IntentVLM—a cognitively inspired, two-stage video-language framework that introduces forward-inverse modeling to open-vocabulary intention recognition for the first time. In the first stage, the model generates candidate goals; in the second, it performs structured selection-based reasoning to interpret intentions. This approach effectively mitigates hallucination in implicit reasoning and circumvents catastrophic forgetting. Evaluated on the IntentQA and Inst-IT Bench datasets, IntentVLM achieves up to 80% accuracy—surpassing baseline methods by a substantial 30% margin and approaching human-level performance.

Technology Category

• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Safety and Robustness

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Improving the effectiveness of human-robot interaction requires social robots to accurately infer human goals through robust intention understanding. This challenge is particularly critical in multimodal settings, where agents must integrate heterogeneous signals including text, visual cues to form a coherent interpretation of user intent. This paper presents IntentVLM, a novel two-stage video-language framework designed for open-vocabulary human intention recognition. The approach is inspired by forward-inverse modeling in cognitive science by decomposing intention understanding into goal candidate generation followed by structured inference through selection, effectively reducing hallucinations in latent reasoning. Evaluated on the IntentQA and Inst-IT Bench datasets, IntentVLM achieves state-of-the-art results with up to 80% accuracy, notably surpassing the baseline performance by 30% and matches human performance. Our findings demonstrate that this structured reasoning approach enhances open-vocabulary intention understanding without catastrophic forgetting, offering a robust foundation for human-centered robotics.

Problem

• intention recognition
• open-vocabulary
• human-robot interaction
• multimodal understanding
• video-language models

Innovation

• open-vocabulary intention recognition
• forward-inverse modeling
• video-language models
• structured reasoning
• human-robot interaction