On Large Uni- and Multi-modal Models for Unsupervised Classification of Social Media Images: Nature's Contribution to People as a case study

2024-09-30

arXiv.org

AI Summary

This study addresses the unsupervised clustering of open-world, abstract semantic concepts—such as cultural ecosystem services—in social media images. We present the first systematic evaluation of large vision models (LVMs), vision-language models (VLMs), and large language models (LLMs) for zero-label semantic clustering. Methodologically, we compare DINOv2 (LVM), LLaVA-1.5 and GPT-4 (VLMs), fine-tuned Qwen (LLM), 10-shot adaptation, and contrastive learning. Results show that fully unsupervised VLM-based clustering achieves >84% accuracy, substantially outperforming conventional unsupervised approaches; fine-tuned DINOv2 and the LLaVA-1.5+LLM hybrid attain >95% accuracy; and 10-shot DINOv2 reaches 83.99%, matching VLM performance. Crucially, we uncover the strong zero-shot generalization capability of VLMs in semantic clustering of social media imagery—an insight previously unreported. This work establishes a novel paradigm for open-world visual understanding grounded in multimodal foundation models.

Technology Category

• Computer Vision: Large Vision Models
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Social media images have proven to be a valuable source of information for understanding human interactions with important subjects such as cultural heritage, biodiversity, and nature, among others. The task of grouping such images into a number of semantically meaningful clusters without labels is challenging due to the high diversity and complex nature of the visual content in addition to their large volume. On the other hand, recent advances in Large Visual Models (LVMs), Large Language Models (LLMs), and Large Visual Language Models (LVLMs) provide an important opportunity to explore new productive and scalable solutions. This work proposes, analyzes, and compares various approaches based on one or more state-of-the-art LVM, LLM, and LVLM, for mapping social media images into a number of predefined classes. As a case study, we consider the problem of understanding the interactions between humans and nature, also known as Nature's Contribution to People or Cultural Ecosystem Services (CES). Our experiments show that the highest-performing approaches, with accuracy above 95%, still require the creation of a small labeled dataset. These include the fine-tuned LVM DINOv2 and the LVLM LLaVA-1.5 combined with a fine-tuned LLM. The top fully unsupervised approaches, achieving accuracy above 84%, are the LVLMs, specifically the proprietary GPT-4 model and the public LLaVA-1.5 model. Additionally, the LVM DINOv2, when applied in a 10-shot learning setup, delivered competitive results with an accuracy of 83.99%, closely matching the performance of the LVLM LLaVA-1.5.

Problem

• Categorizing diverse social media images into meaningful groups
• Addressing abstract concepts with inconsistent visual patterns
• Lack of public datasets for benchmarking image categorization

Innovation

• Utilizes large models for image categorization
• Introduces FLIPS dataset for human-nature interaction
• Evaluates model combinations for cost and scalability