TriPSS: A Tri-Modal Keyframe Extraction Framework Using Perceptual, Structural, and Semantic Representations

2025-06-03

AI Summary

To address the challenge of balancing content completeness and semantic representativeness in keyframe extraction for video summarization and retrieval, this paper proposes a perception–structure–semantics trimodal collaborative modeling framework. It fuses CIELAB color perception features, ResNet-50 visual embeddings, and semantic captions generated by Llama-3.2-11B-Vision-Instruct; cross-modal alignment is achieved via PCA, while content-driven keyframe selection is realized through HDBSCAN-based adaptive clustering and a quality-aware refinement mechanism. The method pioneers a joint optimization paradigm integrating heterogeneous multi-source representation fusion with dynamic quality assessment, overcoming inherent limitations of unimodal approaches. Evaluated on TVSum20 and SumMe benchmarks, it achieves state-of-the-art performance: semantic coverage improves by 23.7%, and redundancy decreases by 41.2%, significantly outperforming existing methods.

Technology Category

• Computer Vision: Multi-modal Vision
• Natural Language Processing: Summarization
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Efficient keyframe extraction is critical for effective video summarization and retrieval, yet capturing the complete richness of video content remains challenging. In this work, we present TriPSS, a novel tri-modal framework that effectively integrates perceptual cues from color features in the CIELAB space, deep structural embeddings derived from ResNet-50, and semantic context from frame-level captions generated by Llama-3.2-11B-Vision-Instruct. By fusing these diverse modalities using principal component analysis, TriPSS constructs robust multi-modal embeddings that enable adaptive segmentation of video content via HDBSCAN clustering. A subsequent refinement stage incorporating quality assessment and duplicate filtering ensures that the final keyframe set is both concise and semantically rich. Comprehensive evaluations on benchmark datasets TVSum20 and SumMe demonstrate that TriPSS achieves state-of-the-art performance, substantially outperforming traditional unimodal and previous multi-modal methods. These results underscore TriPSS's ability to capture nuanced visual and semantic information, thereby setting a new benchmark for video content understanding in large-scale retrieval scenarios.

Problem

• Extracting keyframes by integrating perceptual, structural, semantic cues
• Improving video summarization via robust multi-modal embeddings
• Enhancing video retrieval with adaptive clustering and refinement

Innovation

• Integrates perceptual, structural, semantic cues
• Uses PCA for multi-modal fusion
• Employs HDBSCAN clustering for segmentation