GeMi: A Graph-based, Multimodal Recommendation System for Narrative Scroll Paintings

2026-02-28

AI Summary

This work addresses the critical gap in digital preservation and intelligent access to narrative scroll paintings—an endangered art form—by proposing the first multimodal recommendation system tailored to this domain. The approach integrates visual and textual features to construct a user–item interaction graph, leveraging graph neural networks (GNNs) for graph embedding and message passing to enable content-aware personalized recommendations. By doing so, the method facilitates structured storage and intelligent discovery of narrative scroll paintings, effectively balancing cultural heritage preservation with enhanced user experience. This study thus fills a significant technical void in AI-driven approaches to safeguarding and transmitting intangible cultural heritage.

Technology Category

• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval
• Humans and AI: Intelligent User Interfaces
• Computer Vision: Multi-modal Vision

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs

Abstract

Recommendation Systems are effective in managing the ever-increasing amount of multimodal data available today and help users discover interesting new items. These systems can handle various media types such as images, text, audio, and video data, and this has made it possible to handle content-based recommendation utilizing features extracted from items while also incorporating user preferences. Graph Neural Network (GNN)-based recommendation systems are a special class of recommendation systems that can handle relationships between items and users, making them particularly attractive for content-based recommendations. Their popularity also stems from the fact that they use advanced machine learning techniques, such as deep learning on graph-structured data, to exploit user-to-item interactions. The nodes in the graph can access higher-order neighbor information along with state-of-the-art vision-language models for processing multimodal content, and there are well-designed algorithms for embedding, message passing, and propagation. In this work, we present the design of a GNN-based recommendation system on a novel data set collected from field research. Designed for an endangered performing art form, the recommendation system uses multimodal content (text and image data) to suggest similar paintings for viewing and purchase. To the best of our knowledge, there is no recommendation system designed for narrative scroll paintings -- our work therefore serves several purposes, including art conservation, a data storage system for endangered art objects, and a state-of-the-art recommendation system that leverages both the novel characteristics of the data and preferences of the user population interested in narrative scroll paintings.

Problem

• narrative scroll paintings
• multimodal recommendation
• art conservation
• endangered art
• graph-based recommendation

Innovation

• Graph Neural Network
• Multimodal Recommendation
• Narrative Scroll Paintings
• Vision-Language Model
• Cultural Heritage Preservation