🤖 AI Summary
To address the challenges of efficient retrieval and browsing amid explosive growth in user-generated video content, existing methods (e.g., Video-LLaMA) suffer from fragmented spatiotemporal modeling, high computational overhead, and parameter redundancy. This paper proposes a lightweight and efficient spatiotemporal-cooperative video summarization framework. We introduce a novel hybrid LoRA-based mixture-of-experts architecture with dynamic expert routing; design a dual-path spatiotemporal adaptive module to jointly optimize temporal dynamics and spatial semantics; and incorporate video–language cross-modal alignment during training. Evaluated on VideoXum and ActivityNet, our method achieves state-of-the-art summarization quality while reducing model parameters by 62% and accelerating inference by 3.1× compared to prior approaches. These advances significantly enhance practicality and scalability for large-scale video understanding.
📝 Abstract
With the exponential growth of user-generated content on video-sharing platforms, the challenge of facilitating efficient searching and browsing of videos has garnered significant attention. To enhance users' ability to swiftly locate and review pertinent videos, the creation of concise and informative video summaries has become increasingly important. Video-llama is an effective tool for generating video summarization, but it cannot effectively unify and optimize the modeling of temporal and spatial features and requires a lot of computational resources and time. Therefore, we propose MiLoRA-ViSum to more efficiently capture complex temporal dynamics and spatial relationships inherent in video data and to control the number of parameters for training. By extending traditional Low-Rank Adaptation (LoRA) into a sophisticated mixture-of-experts paradigm, MiLoRA-ViSum incorporates a dual temporal-spatial adaptation mechanism tailored specifically for video summarization tasks. This approach dynamically integrates specialized LoRA experts, each fine-tuned to address distinct temporal or spatial dimensions. Extensive evaluations of the VideoXum and ActivityNet datasets demonstrate that MiLoRA-ViSum achieves the best summarization performance compared to state-of-the-art models, while maintaining significantly lower computational costs. The proposed mixture-of-experts strategy, combined with the dual adaptation mechanism, highlights the model's potential to enhance video summarization capabilities, particularly in large-scale applications requiring both efficiency and precision.