Media Content Atlas: A Pipeline to Explore and Investigate Multidimensional Media Space using Multimodal LLMs

2025-04-22

AI Summary

Traditional screen usage metrics fail to capture the complexity of real-world media experiences. To address this, we propose Media Content Atlas (MCA), the first large-scale, fine-grained, multimodal content analysis framework designed for authentic screen behavior. MCA integrates multimodal large language models (MLLMs), unsupervised vision–semantics clustering, dynamic topic modeling, interpretable image retrieval, and interactive spatiotemporal visualization to enable frame-level content understanding, semantic organization, and open-ended exploration of millions of smartphone screenshots. Evaluated on 1.12 million screenshots collected over 30 days from 112 adults, MCA achieves 96% clustering relevance and 83% accuracy in semantic content description. By unifying inductive exploration with deductive validation, MCA bridges a critical methodological gap in media research and establishes a new paradigm for hypothesis-driven analysis and intervention design in digital behavioral science.

Technology Category

• Application Domains: Humanities & Computational Social Science
• Data Mining & Knowledge Management: Mining of Visual, Multimedia & Multimodal Data
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• User Modeling, Personalization and Recommendation: Practical large-scale studies of user experience
• Security and Privacy: Large-scale security measurements

Abstract

As digital media use continues to evolve and influence various aspects of life, developing flexible and scalable tools to study complex media experiences is essential. This study introduces the Media Content Atlas (MCA), a novel pipeline designed to help researchers investigate large-scale screen data beyond traditional screen-use metrics. Leveraging multimodal large language models (MLLMs), MCA enables moment-by-moment content analysis, content-based clustering, topic modeling, image retrieval, and interactive visualizations. Evaluated on 1.12 million smartphone screenshots continuously captured during screen use from 112 adults over an entire month, MCA facilitates open-ended exploration and hypothesis generation as well as hypothesis-driven investigations at an unprecedented scale. Expert evaluators underscored its usability and potential for research and intervention design, with clustering results rated 96% relevant and descriptions 83% accurate. By bridging methodological possibilities with domain-specific needs, MCA accelerates both inductive and deductive inquiry, presenting new opportunities for media and HCI research.

Problem

• Develop scalable tools to analyze complex media experiences
• Enable moment-by-moment content analysis using multimodal LLMs
• Facilitate large-scale exploration of screen data for research

Innovation

• Uses multimodal LLMs for content analysis
• Enables clustering and topic modeling
• Supports interactive visualizations exploration