CognArtive: Large Language Models for Automating Art Analysis and Decoding Aesthetic Elements

2025-02-04

AI Summary

This study addresses the lack of scalable, structured formal analysis tools in art history by introducing the first LLM/MLLM-driven framework for artistic formal analysis. Methodologically, it integrates multimodal large language models (MLLMs), domain-knowledge-enhanced prompt engineering, temporal pattern mining, and interactive visualization to enable fine-grained, structured parsing of visual elements, composition, technique, and aesthetic expression in paintings. Key contributions include: (1) the first systematic application of LLMs/MLLMs to formalist art analysis; (2) a novel, interpretable aesthetic decoding paradigm grounded in computational hermeneutics; and (3) scalable automation—processing over one thousand artworks per minute—with 87% accuracy in cross-period stylistic evolution identification, approaching expert-level performance. An open-source, web-based visualization platform is publicly available and operational.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Large Vision Models
• Natural Language Processing: Sentiment Analysis, Stylistic Analysis, and Argument Mining

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Art, as a universal language, can be interpreted in diverse ways, with artworks embodying profound meanings and nuances. The advent of Large Language Models (LLMs) and the availability of Multimodal Large Language Models (MLLMs) raise the question of how these transformative models can be used to assess and interpret the artistic elements of artworks. While research has been conducted in this domain, to the best of our knowledge, a deep and detailed understanding of the technical and expressive features of artworks using LLMs has not been explored. In this study, we investigate the automation of a formal art analysis framework to analyze a high-throughput number of artworks rapidly and examine how their patterns evolve over time. We explore how LLMs can decode artistic expressions, visual elements, composition, and techniques, revealing emerging patterns that develop across periods. Finally, we discuss the strengths and limitations of LLMs in this context, emphasizing their ability to process vast quantities of art-related data and generate insightful interpretations. Due to the exhaustive and granular nature of the results, we have developed interactive data visualizations, available online https://cognartive.github.io/, to enhance understanding and accessibility.

Problem

• Automating art analysis using LLMs
• Decoding aesthetic elements in artworks
• Exploring LLM strengths in art interpretation

Innovation

• LLMs automate art analysis
• MLLMs decode aesthetic elements
• Interactive visualizations enhance understanding