Multimodal Large Models Are Effective Action Anticipators

2025-01-01

AI Summary

To address the challenges of temporal modeling and semantic understanding in long-horizon video action anticipation, this paper proposes ActionLLM—a novel framework that encodes video frame sequences into text-like tokens and pioneers the integration of large language models (LLMs) into action anticipation. Methodologically, ActionLLM introduces: (i) a lightweight cross-modal interaction module for vision-language alignment; (ii) a streamlined LLM architecture jointly optimized with an action-specific linear decoder, enabling end-to-end action classification without instruction tuning; and (iii) a sequential video tokenization strategy. Experiments on multiple benchmarks demonstrate substantial improvements over RNN- and Transformer-based approaches, validating the effectiveness, generalizability, and deployment feasibility of LLMs for long-horizon action anticipation.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Large Vision Models
• Natural Language Processing: (Large) Language Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

The task of long-term action anticipation demands solutions that can effectively model temporal dynamics over extended periods while deeply understanding the inherent semantics of actions. Traditional approaches, which primarily rely on recurrent units or Transformer layers to capture long-term dependencies, often fall short in addressing these challenges. Large Language Models (LLMs), with their robust sequential modeling capabilities and extensive commonsense knowledge, present new opportunities for long-term action anticipation. In this work, we introduce the ActionLLM framework, a novel approach that treats video sequences as successive tokens, leveraging LLMs to anticipate future actions. Our baseline model simplifies the LLM architecture by setting future tokens, incorporating an action tuning module, and reducing the textual decoder layer to a linear layer, enabling straightforward action prediction without the need for complex instructions or redundant descriptions. To further harness the commonsense reasoning of LLMs, we predict action categories for observed frames and use sequential textual clues to guide semantic understanding. In addition, we introduce a Cross-Modality Interaction Block, designed to explore the specificity within each modality and capture interactions between vision and textual modalities, thereby enhancing multimodal tuning. Extensive experiments on benchmark datasets demonstrate the superiority of the proposed ActionLLM framework, encouraging a promising direction to explore LLMs in the context of action anticipation. Code is available at https://github.com/2tianyao1/ActionLLM.git.

Problem

• Long-term behavior prediction
• Video action prediction
• Temporal understanding

Innovation

• ActionLLM
• cross-modal interaction
• long-term action prediction