video

To address insufficient prediction accuracy of video frame forecasting in critical applications such as weather forecasting and autonomous driving, this paper proposes a hybrid deep architecture integrating an autoencoder with spatiotemporal modeling networks. We systematically evaluate the performance of autoencoders cascaded with RNNs, 3D CNNs, and ConvLSTMs, identifying the autoencoder–3D CNN–ConvLSTM pipeline as optimal for real-world grayscale videos. Using SSIM as a unified evaluation metric, we validate our approach on three heterogeneous video datasets, achieving an average SSIM improvement from 0.69 to 0.82—significantly outperforming single-model baselines. Our work elucidates the synergistic gains arising from multi-stage feature compression and sequential modeling, and establishes a reproducible, high-fidelity benchmark framework for video prediction in complex dynamic scenes.

To address insufficient spatiotemporal modeling capabilities amid explosive growth in video content, this paper proposes a unified spatiotemporal analysis framework that jointly enhances long-range dependency modeling through multi-scale temporal modeling and dynamic spatial attention. Methodologically, the framework systematically integrates 3D CNNs, Transformer-based video encoders, and contrastive learning pretraining, and is comprehensively evaluated across multiple benchmarks—including Kinetics, Something-Something, and UCF101. The core contribution lies in the decoupled yet synergistic optimization of spatial and temporal representations: dynamic attention adaptively focuses on salient frames and regions, while multi-scale temporal modules hierarchically capture both short-term motion patterns and long-term semantic structures. Experiments demonstrate an average 4.2% improvement in action recognition accuracy, an 18% reduction in temporal localization error, and concurrent gains in model generalizability and interpretability.

Existing video compression methods are primarily optimized for human visual perception and thus often fail to preserve semantic information critical for machine vision tasks. To address this, this paper proposes a machine-vision-oriented neural preprocessing framework. It introduces a learnable preprocessor prior to standard video encoding and pioneers a differentiable virtual codec, enabling end-to-end joint optimization of preprocessing and conventional encoders (e.g., H.264/AVC) without modifying codec standards. A rate–distortion–task loss jointly optimizes bit rate, reconstruction fidelity, and downstream task performance—including object detection and action recognition. Experiments demonstrate that the framework reduces average bit rate by over 15% while maintaining or even improving task accuracy, significantly enhancing semantic fidelity and utility of compressed video for machine vision applications.

Real-time video analysis faces a fundamental trade-off between spatiotemporal modeling accuracy and inference efficiency, particularly under resource constraints. To address this, we propose a unified multi-task framework that jointly performs action recognition and object tracking. Our approach employs a hierarchical attention mechanism to adaptively focus on temporally salient spatial regions and introduces parallel sequence modeling to enhance computational efficiency. By integrating advanced spatiotemporal representation learning with lightweight co-optimization strategies, the method achieves state-of-the-art performance: +3.2% and +2.8% top-1 accuracy on UCF-101 and HMDB-51 for action recognition, +2.8% MOTA on MOT17 for multi-object tracking, and a 40% speedup in end-to-end inference latency—significantly outperforming existing real-time approaches.

The absence of a unified evaluation benchmark for video compression optimized for downstream vision tasks (e.g., detection, recognition) hinders standardization of Feature Coding for Machines (FCM). Method: This paper introduces the first open-source evaluation platform supporting joint optimization across multiple dimensions: vision tasks (image/video understanding), model architectures (CNNs/Transformers), and codecs (traditional and deep learning-based). It proposes a novel rate-distortion–task-accuracy joint analysis framework, integrating standardized codecs with differentiable neural compression modules to enable end-to-end co-evaluation of compression efficiency and downstream task performance. Contribution/Results: Extensive validation on multiple benchmark datasets demonstrates significant improvements in task-driven compression efficiency. The platform has been formally adopted by MPEG as the core infrastructure for FCM standard development and evaluation.

Current video large language models (Video-LLMs) face significant bottlenecks in modeling complex temporal dynamics—such as action evolution and inter-frame dependencies—hindering fine-grained temporal reasoning. To address this, we propose a novel architecture that pioneers the integration of stacked temporal attention modules directly into the visual encoder, enabling explicit time-structure and action-sequence modeling at the early visual representation stage. This design synergizes with a multimodal fusion mechanism to enhance inter-frame relational modeling and cross-modal alignment. Evaluated on VITATECS, MVBench, and Video-MME benchmarks, our model achieves an average performance gain of +5.5%, with particularly strong improvements in action recognition and temporal question answering—outperforming state-of-the-art methods. Our core contribution lies in fundamentally shifting temporal modeling to the foundational layer of the visual encoder, thereby overcoming the limitations of conventional Video-LLMs that rely solely on post-hoc fusion or lightweight temporal modules.

This study investigates the impact of various attention mechanisms on the performance of 3D video classification models under a setting where spatial resolution is enhanced at the expense of temporal cues. Building upon three canonical 3D CNN architectures—MC3, R3D, and R(2+1)D—the authors introduce Dropout layers to simulate scenarios with limited temporal information and systematically integrate ten attention modules, including CBAM, TCN, multi-head attention, and channel attention, for comparative evaluation. Experiments on the UCF101 dataset demonstrate that an enhanced R(2+1)D model combined with multi-head attention achieves 88.98% accuracy, while revealing substantial performance variations across different attention mechanisms at the class level. This work provides the first systematic analysis of how degraded temporal features critically affect 3D action recognition, offering novel insights for modeling high-resolution videos with weak temporal signals.

This work proposes treating temporal flow rate as a learnable visual concept to enable perception and controllable generation of video playback speed. Through a self-supervised approach leveraging multimodal cues and temporal structure inherent in videos, the model accurately detects speed variations and estimates actual playback rates. Building upon this framework, the authors construct the largest wild slow-motion video dataset to date and demonstrate two key applications: synthesizing videos at user-specified playback speeds and performing temporal super-resolution on low-frame-rate videos to recover fine-grained dynamic details. This study is the first to model time flow rate as a manipulable perceptual dimension, significantly advancing capabilities in video temporal understanding and generation.

This work proposes a lightweight compressed-domain tracking model that addresses the high computational cost and limited real-time performance of conventional object tracking methods in large-scale video surveillance, which typically require full decoding of RGB video streams. By directly leveraging motion vectors and transform coefficients from compressed bitstreams without decoding the original video, the proposed approach enables cross-frame propagation of object bounding boxes for the first time. Built upon codec-domain modeling, the method substantially reduces computational overhead while maintaining high accuracy—achieving only a 4% drop in mAP@0.5 compared to an RGB-based baseline on the MOTS15/17/20 benchmarks, with up to a 3.7× speedup in inference throughput.

To address inefficiency and challenges in modeling long-range temporal dependencies in long-video understanding, this paper proposes a hybrid Mamba-Transformer architecture: Mamba efficiently captures global temporal dynamics, while Transformer enhances local fine-grained interactions. We introduce the TransV module, which enables directed information compression and transfer from visual tokens to instruction tokens, significantly mitigating visual redundancy. Furthermore, we unify the visual encoder with the large language model to improve cross-modal alignment and temporal reasoning. Our method achieves state-of-the-art performance across multiple long-video understanding benchmarks, supports end-to-end processing of videos exceeding 10,000 frames (hour-scale), and extends frame capacity by 2–3× over existing approaches. Notably, we uncover a dynamic division of labor between attention mechanisms within the hybrid architecture—revealing how Mamba and Transformer complementarily specialize across temporal scales. This work establishes a novel paradigm for efficient multimodal modeling of long-duration video.