Unsupervised Multimodal Graph-based Model for Geo-social Analysis

2025-11-26

AI Summary

Social media multimodal data (text + geolocation) are often processed in isolation, lacking end-to-end joint modeling for geographic social analysis. Method: This paper proposes an unsupervised multimodal graph representation framework tailored for geographic social analysis. It innovatively introduces a dual-architecture design—MonoGraph and MultiGraph—that integrates graph neural networks, multi-head attention, and contrastive learning to jointly embed textual and geospatial modalities. A composite loss function—combining contrastive, consistency, and alignment objectives—enables deep multimodal fusion under purely unsupervised settings. Contribution/Results: Evaluated on four real-world disaster datasets, the framework significantly improves event topic coherence, spatial consistency, and interpretability. It further demonstrates strong cross-domain generalization capability, advancing unsupervised learning for geotagged social media analytics.

Technology Category

• Machine Learning: Multimodal Learning
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Computer Vision: Multi-modal Vision

Application Category

• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• Social Networks and Social Media: Social media analysis through the lenses of networks

Abstract

The systematic analysis of user-generated social media content, especially when enriched with geospatial context, plays a vital role in domains such as disaster management and public opinion monitoring. Although multimodal approaches have made significant progress, most existing models remain fragmented, processing each modality separately rather than integrating them into a unified end-to-end model. To address this, we propose an unsupervised, multimodal graph-based methodology that jointly embeds semantic and geographic information into a shared representation space. The proposed methodology comprises two architectural paradigms: a mono graph (MonoGrah) model that jointly encodes both modalities, and a multi graph (MultiGraph) model that separately models semantic and geographic relationships and subsequently integrates them through multi-head attention mechanisms. A composite loss, combining contrastive, coherence, and alignment objectives, guides the learning process to produce semantically coherent and spatially compact clusters. Experiments on four real-world disaster datasets demonstrate that our models consistently outperform existing baselines in topic quality, spatial coherence, and interpretability. Inherently domain-independent, the framework can be readily extended to diverse forms of multimodal data and a wide range of downstream analysis tasks.

Problem

• Integrates semantic and geographic data into a unified model
• Addresses fragmented multimodal analysis in social media content
• Enhances topic quality and spatial coherence for disaster datasets

Innovation

• Unsupervised multimodal graph model integrates semantic and geographic data
• Uses MonoGraph and MultiGraph with attention for joint embedding
• Composite loss function ensures coherent and compact clustering