Is Your Image a Good Storyteller?

2024-12-29

AI Summary

This paper addresses the long-overlooked problem of quantifying image semantic complexity by introducing the Image Semantic Assessment (ISA) task—designed to automatically evaluate an image’s capacity to evoke vivid, narrative-rich interpretations, thereby supporting cognitive assessment, cross-cultural visual content generation, and large multimodal model training. We present the first ISA benchmark dataset and innovatively leverage large language models (LLMs) as proxy evaluators of visual semantic complexity, enabling fully automated, zero-shot, human-annotation-free scoring. Our method integrates CLIP with LLMs via cross-modal alignment, prompt engineering, and semantic consistency distillation to enhance robustness. On the ISA benchmark, our approach significantly outperforms purely vision-based models and achieves high agreement with human experts (Spearman ρ > 0.82). Both code and dataset are publicly released.

Technology Category

• Computer Vision: Language and Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Large language models for search
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Quantifying image complexity at the entity level is straightforward, but the assessment of semantic complexity has been largely overlooked. In fact, there are differences in semantic complexity across images. Images with richer semantics can tell vivid and engaging stories and offer a wide range of application scenarios. For example, the Cookie Theft picture is such a kind of image and is widely used to assess human language and cognitive abilities due to its higher semantic complexity. Additionally, semantically rich images can benefit the development of vision models, as images with limited semantics are becoming less challenging for them. However, such images are scarce, highlighting the need for a greater number of them. For instance, there is a need for more images like Cookie Theft to cater to people from different cultural backgrounds and eras. Assessing semantic complexity requires human experts and empirical evidence. Automatic evaluation of how semantically rich an image will be the first step of mining or generating more images with rich semantics, and benefit human cognitive assessment, Artificial Intelligence, and various other applications. In response, we propose the Image Semantic Assessment (ISA) task to address this problem. We introduce the first ISA dataset and a novel method that leverages language to solve this vision problem. Experiments on our dataset demonstrate the effectiveness of our approach. Our data and code are available at: https://github.com/xiujiesong/ISA.

Problem

• Semantic Complexity
• Image Analysis
• Automatic Evaluation

Innovation

• Image Semantic Assessment
• Semantic Complexity Evaluation
• Automated Cognitive Assessment