ExpArt-KG: Artwork Image Description Generation through Iterative Exploration of Knowledge Graphs

2026-08-31

AI Summary

为解决LVLMs难以准确描述图像中实体关系的问题，本文提出一种结合知识图谱迭代检索与生成的方法，并构建艺术作品领域知识图谱ExpArt-KG。

Abstract

Large Vision-Language Models (LVLMs) achieve strong performance on image-grounded text generation and visual question answering. However, it remains difficult for them to comprehensively and accurately describe the factual relations among the entities and concepts associated with the objects depicted in an image. In this work, we propose a framework that efficiently exploits factual information from a knowledge graph via retrieval-augmented generation (RAG), with the goal of enabling LVLMs to generate detailed and accurate image explanations. Specifically, our method alternates between answer generation and knowledge-graph retrieval, and controls the search using a correctness judgment, thereby acquiring the necessary and sufficient factual information efficiently. We also construct a knowledge graph for the artwork domain (ExpArt-KG), in which the correspondence between images and entities is unambiguous. Applying the proposed method to this knowledge graph, we show experimentally that it improves the level of detail of artwork explanations and reduces the retrieval cost of external knowledge while maintaining generation quality comparable to that of iterating a fixed number of times.

Problem

• Large Vision-Language Models
• Image-grounded Text Generation
• Factual Relations
• Knowledge Graphs
• Artwork Explanation

Innovation

• retrieval-augmented generation
• knowledge graph
• iterative exploration
• artwork image description
• correctness judgment