🤖 AI Summary
This work addresses the challenge of domain shift that hinders mainstream image captioning models from accurately capturing the nuanced cultural semantics embedded in Dongba paintings. To this end, the authors propose the PVGF-DPC framework, which integrates a culture-aware content prompting mechanism to steer caption generation and introduces a vision-semantic–generation fusion loss to align visual features with culturally specific themes. The model leverages MobileNetV2 for visual feature extraction and a 10-layer Transformer decoder initialized with BERT, enhanced by prompt-guided decoding and joint optimization. Evaluated on a newly constructed dataset of 9,408 augmented Dongba images, the approach demonstrates significant improvements in both semantic accuracy and cultural relevance across seven Dongba thematic categories.
📝 Abstract
Dongba paintings, the treasured pictorial legacy of the Naxi people in southwestern China, feature richly layered visual elements, vivid color palettes, and pronounced ethnic and regional cultural symbolism, yet their automatic textual description remains largely unexplored owing to severe domain shift when mainstream captioning models are applied directly. This paper proposes \textbf{PVGF-DPC} (\textit{Prompt and Visual Semantic-Generation Fusion-based Dongba Painting Captioning}), an encoder-decoder framework that integrates a content prompt module with a novel visual semantic-generation fusion loss to bridge the gap between generic natural-image captioning and the culturally specific imagery found in Dongba art. A MobileNetV2 encoder extracts discriminative visual features, which are injected into the layer normalization of a 10-layer Transformer decoder initialized with pretrained BERT weights; meanwhile, the content prompt module maps the image feature vector to culture-aware labels -- such as \emph{deity}, \emph{ritual pattern}, or \emph{hell ghost} -- and constructs a post-prompt that steers the decoder toward thematically accurate descriptions. The visual semantic-generation fusion loss jointly optimizes the cross-entropy objectives of both the prompt predictor and the caption generator, encouraging the model to extract key cultural and visual cues and to produce captions that are semantically aligned with the input image. We construct a dedicated Dongba painting captioning dataset comprising 9{}408 augmented images with culturally grounded annotations spanning seven thematic categories.