Visual Chronicles: Using Multimodal LLMs to Analyze Massive Collections of Images

2025-04-11

AI Summary

This work addresses the challenge of open-ended visual trend mining in ultra-large-scale urban temporal image repositories (tens of millions of images). We propose the first unsupervised spatiotemporal trend discovery framework leveraging multimodal large language models (MLLMs). Methodologically, we overcome MLLM context-length limitations via a bottom-up hierarchical semantic abstraction and temporal clustering pipeline, integrating zero-shot prompt engineering with cross-temporal image semantic alignment—enabling category- and annotation-free trend induction. Key contributions include: (1) the first application of MLLMs to unsupervised, open-domain visual trend mining at billion-image scale; (2) a scalable spatiotemporal decomposition architecture; and (3) superior performance on real-world urban data over baselines, automatically identifying fine-grained, interpretable, and interactively verifiable long-term patterns—e.g., “installation of outdoor dining areas.”

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Large Vision Models
• Data Mining & Knowledge Management: Mining of Visual, Multimedia & Multimodal Data

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

We present a system using Multimodal LLMs (MLLMs) to analyze a large database with tens of millions of images captured at different times, with the aim of discovering patterns in temporal changes. Specifically, we aim to capture frequent co-occurring changes ("trends") across a city over a certain period. Unlike previous visual analyses, our analysis answers open-ended queries (e.g.,"what are the frequent types of changes in the city?") without any predetermined target subjects or training labels. These properties cast prior learning-based or unsupervised visual analysis tools unsuitable. We identify MLLMs as a novel tool for their open-ended semantic understanding capabilities. Yet, our datasets are four orders of magnitude too large for an MLLM to ingest as context. So we introduce a bottom-up procedure that decomposes the massive visual analysis problem into more tractable sub-problems. We carefully design MLLM-based solutions to each sub-problem. During experiments and ablation studies with our system, we find it significantly outperforms baselines and is able to discover interesting trends from images captured in large cities (e.g.,"addition of outdoor dining,","overpass was painted blue,"etc.). See more results and interactive demos at https://boyangdeng.com/visual-chronicles.

Problem

• Analyzing massive image collections for temporal change patterns
• Discovering frequent co-occurring urban changes without predefined labels
• Scaling MLLMs for large datasets via bottom-up sub-problem decomposition

Innovation

• Multimodal LLMs analyze temporal image patterns
• Open-ended queries without predefined labels
• Bottom-up decomposition for large datasets