VisTCP: A Visualization Framework to Construct Knowledge-Graph-Based Representation for Traditional Chinese Painting

2026-07-07

AI Summary

This study addresses the challenge of accurately capturing the semantics of traditional Chinese paintings, which differ significantly from modern imagery and involve difficult-to-recognize historical artifacts, rendering existing structured representation methods inadequate. To overcome this, the work proposes the first human-in-the-loop framework that integrates art-historical expert knowledge with visual feedback. The approach combines an expert-driven semantic taxonomy, a painting-specific structured representation model, and joint embedding visualization techniques to enable interpretable and revisable semantic modeling. Through case studies, usage scenario analyses, and expert interviews, the framework demonstrates substantial improvements in both accuracy and usability of structured representations for traditional Chinese paintings, effectively supporting scholars in semantic interpretation and iterative refinement.

Abstract

Structured representation can characterize semantic objects and relationships in images. It provides a possible effective way for the semantic understanding of Traditional Chinese Paintings (TCPs) to better support archaeology and art history research. However, most image-oriented structured representation methods perform poorly on TCPs, due to two major challenges: 1) the objects and events of TCPs exhibit substantial differences from modern natural images, which results in semantic misunderstandings of TCPs; and 2) it is difficult to achieve accurate identification of ancient objects and events in TCPs, even for domain experts.In this paper, we propose VisTCP, a visualization framework that combines a TCP-oriented intelligent model and expert knowledge, which enables art historians to achieve trustworthy structured representations of TCPs in a human-in-the-loop manner. Firstly, we conduct a pilot study with three domain experts to build a semantic taxonomy of TCPs. Then, expert-annotated data are used to train a TCP-oriented structured representation model, which can automatically extract meaningful objects and their relationships in TCPs. To inform users of the model uncertainty, we design a joint embedding visualization view to show the differences between expert annotations and model predictions. This allows users to refine the structured representation based on their domain knowledge, enabling iterative optimization of the model. Finally, we conduct a case study, a usage scenario, and expert interviews on a real dataset to demonstrate the effectiveness of VisTCP in supporting the structured representation and semantic understanding of TCPs.

Problem

• Traditional Chinese Painting
• structured representation
• semantic understanding
• object identification
• domain knowledge

Innovation

• knowledge graph
• structured representation
• human-in-the-loop
• traditional Chinese painting
• joint embedding visualization