Exploring High-Order Self-Similarity for Video Understanding

2026-04-22

AI Summary

Existing video understanding methods struggle to effectively model complex temporal dynamics. To address this limitation, this work presents the first systematic exploration of high-order spatiotemporal self-similarity (STSS) and introduces a lightweight, general-purpose Multi-Order Self-Similarity (MOSS) module. MOSS enhances action modeling by learning and fusing STSS features across multiple orders. Implemented with neural networks for efficient feature extraction and integration, the proposed module consistently achieves significant performance gains across diverse benchmarks, including action recognition, motion-oriented video question answering, and real-world robotic tasks, thereby demonstrating its broad applicability and effectiveness.

Technology Category

• Computer Vision: Motion & Tracking
• Machine Learning: Mixture of Experts (MoE)
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for heterogeneous, signed, attributed, multi-relational, temporal, higher-order, and annotated Web-related graphs
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Search and Retrieval-Augmented AI: Web query analysis, representation and understanding

Abstract

Space-time self-similarity (STSS), which captures visual correspondences across frames, provides an effective way to represent temporal dynamics for video understanding. In this work, we explore higher-order STSS and demonstrate how STSSs at different orders reveal distinct aspects of these dynamics. We then introduce the Multi-Order Self-Similarity (MOSS) module, a lightweight neural module designed to learn and integrate multi-order STSS features. It can be applied to diverse video tasks to enhance motion modeling capabilities while consuming only marginal computational cost and memory usage. Extensive experiments on video action recognition, motion-centric video VQA, and real-world robotic tasks consistently demonstrate substantial improvements, validating the broad applicability of MOSS as a general temporal modeling module. The source code and checkpoints will be publicly available.

Problem

• self-similarity
• video understanding
• temporal dynamics
• high-order
• motion modeling

Innovation

• Multi-Order Self-Similarity
• Space-Time Self-Similarity
• Temporal Modeling
• Video Understanding
• Lightweight Neural Module