An Adaptive Data-Resilient Multi-Modal Framework for Hierarchical Multi-Label Book Genre Identification

2025-05-05

AI Summary

This paper addresses robust multi-label genre classification for books under incomplete or noisy multimodal inputs—namely, book covers, summaries, metadata, and full-text content. We propose a dynamic modality-adaptive selection mechanism coupled with a hierarchical multi-label joint modeling framework, integrating ViT, BERT, and MLP encoders. Our approach incorporates gated modality weighting and a hierarchical label decoder, along with a missingness-robust training strategy. Key contributions include: (1) the first publicly available hierarchical book genre annotation dataset; (2) stable inference under extreme missingness scenarios—including zero-text and zero-image inputs; and (3) state-of-the-art performance, achieving an average F1-score improvement of 12.6% under multimodal missingness and outperforming prior work by 9.3% in hierarchical accuracy.

Technology Category

• Machine Learning: Multimodal Learning
• Computer Vision: Multi-modal Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

Identifying the finer details of a book's genres enhances user experience by enabling efficient book discovery and personalized recommendations, ultimately improving reader engagement and satisfaction. It also provides valuable insights into market trends and consumer preferences, allowing publishers and marketers to make data-driven decisions regarding book production and marketing strategies. While traditional book genre classification methods primarily rely on review data or textual analysis, incorporating additional modalities, such as book covers, blurbs, and metadata, can offer richer context and improve prediction accuracy. However, the presence of incomplete or noisy information across these modalities presents a significant challenge. This paper introduces IMAGINE (Intelligent Multi-modal Adaptive Genre Identification NEtwork), a framework designed to address these complexities. IMAGINE extracts robust feature representations from multiple modalities and dynamically selects the most informative sources based on data availability. It employs a hierarchical classification strategy to capture genre relationships and remains adaptable to varying input conditions. Additionally, we curate a hierarchical genre classification dataset that structures genres into a well-defined taxonomy, accommodating the diverse nature of literary works. IMAGINE integrates information from multiple sources and assigns multiple genre labels to each book, ensuring a more comprehensive classification. A key feature of our framework is its resilience to incomplete data, enabling accurate predictions even when certain modalities, such as text, images, or metadata, are missing or incomplete. Experimental results show that IMAGINE outperformed existing baselines in genre classification accuracy, particularly in scenarios with insufficient modality-specific data.

Problem

• Enhances book genre identification using multi-modal data
• Addresses incomplete or noisy information across modalities
• Improves classification accuracy with adaptive data resilience

Innovation

• Multi-modal feature extraction for genre identification
• Dynamic data source selection based on availability
• Hierarchical classification with resilience to incomplete data