Decoupled Object-Centric Video Understanding for Generating Robotic Manipulation Commands

2026-06-15

AI Summary

Existing methods struggle to accurately identify objects in videos that are functionally relevant to actions, often yielding robot instructions that are semantically plausible but operationally ambiguous. To address this, this work proposes a decoupled framework for action recognition and object selection, which explicitly separates action understanding from object localization for the first time. The approach introduces trajectory role classification, ambiguity detection, and overlap minimization strategies to precisely filter task-relevant objects, and integrates a vision-language model to generate syntactically correct manipulation instructions. Evaluated on a modified Something-Something V2 dataset, the method achieves an action classification accuracy of 86.79% and BLEU-4 scores of 0.337 and 0.261 on known and novel objects, respectively, with METEOR and CIDEr metrics improving by up to 171.7% over baselines, demonstrating strong zero-shot generalization capability.

Abstract

Translating video demonstrations into executable robot commands remains challenging because existing methods often fail to identify which objects are functionally involved in the demonstrated action. As a result, they may generate commands that are linguistically plausible but operationally ambiguous. We propose an object-centric video understanding framework that decouples action recognition from object identification to generate precise, grammar-free manipulation commands. Our approach integrates Temporal Shift Modules (TSM) for efficient spatio-temporal action classification with a novel \textbf{Object Selection} algorithm that identifies task-relevant objects through trajectory-based role classification, blur detection, and overlap minimization. The selected objects are then processed by Vision-Language Models (VLMs) for robust category recognition and zero-shot generalization. Evaluated on a modified Something-Something V2 dataset, our method achieves 86.79\% action classification accuracy and BLEU-4 scores of 0.337 on standard objects and 0.261 on novel objects. These results improve over the strongest task-specific baseline by 80.2\% and 143.9\%, respectively. Larger gains are observed in METEOR and CIDEr, reaching 157.9\% and 171.7\% on novel objects. Across all semantic metrics, our approach consistently outperforms task-specific methods and remains competitive with, or surpasses, large general-purpose VLMs while retaining a modular, object-centric design.

Problem

• object-centric
• robotic manipulation
• video understanding
• action recognition
• object identification

Innovation

• Object-Centric
• Decoupled Action Recognition
• Object Selection
• Temporal Shift Modules
• Vision-Language Models