InstrAct: Towards Action-Centric Understanding in Instructional Videos

2026-04-09

AI Summary

Existing video foundation models struggle to capture fine-grained actions and temporal relationships in instructional videos due to supervision noise and static biases. This work proposes InstrAct, the first framework to systematically develop action-centric representation learning for instructional video understanding. InstrAct employs data-driven noisy caption filtering and generates action-centric hard negatives, integrates an action-aware motion extractor, and introduces Dynamic Time Warping Alignment (DTW-Align) alongside Masked Action Modeling (MAM) to enhance learning of action semantics and temporal structure. Evaluated on the newly curated InstrAct Bench, the proposed method significantly outperforms state-of-the-art models across semantic reasoning, procedural logic comprehension, and fine-grained retrieval tasks.

Technology Category

• Computer Vision: Video Understanding & Activity Analysis
• Knowledge Representation and Reasoning: Action, Change, and Causality
• Machine Learning: Representation Learning

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Understanding instructional videos requires recognizing fine-grained actions and modeling their temporal relations, which remains challenging for current Video Foundation Models (VFMs). This difficulty stems from noisy web supervision and a pervasive"static bias", where models rely on objects rather than motion cues. To address this, we propose InstrAction, a pretraining framework for instructional videos'action-centric representations. We first introduce a data-driven strategy, which filters noisy captions and generates action-centric hard negatives to disentangle actions from objects during contrastive learning. At the visual feature level, an Action Perceiver extracts motion-relevant tokens from redundant video encodings. Beyond contrastive learning, we introduce two auxiliary objectives: Dynamic Time Warping alignment (DTW-Align) for modeling sequential temporal structure, and Masked Action Modeling (MAM) for strengthening cross-modal grounding. Finally, we introduce the InstrAct Bench to evaluate action-centric understanding, where our method consistently outperforms state-of-the-art VFMs on semantic reasoning, procedural logic, and fine-grained retrieval tasks.

Problem

• instructional videos
• action understanding
• static bias
• temporal relations
• fine-grained actions

Innovation

• action-centric representation
• contrastive learning with hard negatives
• Action Perceiver
• Dynamic Time Warping alignment
• Masked Action Modeling