Cross-Modal Prototype Augmentation and Dual-Grained Prompt Learning for Social Media Popularity Prediction

2025-08-22

AI Summary

To address insufficient vision–text alignment and superficial multimodal fusion in social media popularity prediction, this paper proposes a Multi-level Prototype-enhanced Framework. The method innovatively integrates cross-modal attention—enhancing semantic alignment and structural modeling between images and text—with dual-granularity prompt learning: coarse-grained (category-level) and fine-grained (instance-level) prompts jointly optimize modality consistency and hierarchical association discovery. Additionally, a contrastive learning–driven hierarchical prototype network is introduced to improve class discriminability and cross-modal representation robustness. Evaluated on multiple mainstream benchmarks, the framework consistently outperforms existing state-of-the-art methods, achieving average accuracy gains of 3.2%–5.7%. It establishes a novel, interpretable, and scalable paradigm for multimodal social media analysis.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• Social Networks and Social Media: Social media analysis through the lenses of networks
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG

Abstract

Social Media Popularity Prediction is a complex multimodal task that requires effective integration of images, text, and structured information. However, current approaches suffer from inadequate visual-textual alignment and fail to capture the inherent cross-content correlations and hierarchical patterns in social media data. To overcome these limitations, we establish a multi-class framework , introducing hierarchical prototypes for structural enhancement and contrastive learning for improved vision-text alignment. Furthermore, we propose a feature-enhanced framework integrating dual-grained prompt learning and cross-modal attention mechanisms, achieving precise multimodal representation through fine-grained category modeling. Experimental results demonstrate state-of-the-art performance on benchmark metrics, establishing new reference standards for multimodal social media analysis.

Problem

• Inadequate visual-textual alignment in multimodal data
• Failure to capture cross-content correlations and hierarchies
• Need for precise multimodal representation through category modeling

Innovation

• Hierarchical prototypes with contrastive learning for alignment
• Dual-grained prompt learning for category modeling
• Cross-modal attention mechanisms for precise representation