ArtContext: Contextualizing Artworks with Open-Access Art History Articles and Wikidata Knowledge through a LoRA-Tuned CLIP Model

2026-02-11

AI Summary

This study addresses the challenge that existing open-access art historical texts are poorly linked to specific artworks and their fine-grained visual features—such as composition, iconography, and material culture—thereby hindering efficient access to scholarly context. To bridge this gap, the authors propose the first weakly supervised multimodal alignment framework that integrates art historical narratives with structured knowledge from Wikidata. They develop an end-to-end pipeline that combines a custom-built corpus collection system with knowledge base enrichment and employs LoRA-based fine-tuning to adapt the CLIP model, yielding a specialized variant termed PaintingCLIP. Experimental results demonstrate that this approach significantly outperforms the original CLIP on artwork contextual understanding tasks, delivering accurate and granular scholarly annotations. The framework also exhibits strong potential for generalization to other domains within the humanities.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Web Mining and Content Analysis: Large pretrained models with web data
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Many Art History articles discuss artworks in general as well as specific parts of works, such as layout, iconography, or material culture. However, when viewing an artwork, it is not trivial to identify what different articles have said about the piece. Therefore, we propose ArtContext, a pipeline for taking a corpus of Open-Access Art History articles and Wikidata Knowledge and annotating Artworks with this information. We do this using a novel corpus collection pipeline, then learn a bespoke CLIP model adapted using Low-Rank Adaptation (LoRA) to make it domain-specific. We show that the new model, PaintingCLIP, which is weakly supervised by the collected corpus, outperforms CLIP and provides context for a given artwork. The proposed pipeline is generalisable and can be readily applied to numerous humanities areas.

Problem

• artwork contextualization
• art history articles
• Wikidata knowledge
• information retrieval
• art annotation

Innovation

• LoRA-tuned CLIP
• ArtContext
• open-access art history
• weakly supervised learning
• multimodal knowledge integration