Building a Multimodal Dataset of Academic Paper for Keyword Extraction

2026-06-29

AI Summary

This study addresses the limitation of existing keyword extraction methods, which predominantly focus on plain text while neglecting visual and audio modalities, and the absence of dedicated multimodal datasets. To bridge this gap, the authors introduce the first multimodal dataset comprising 1,000 academic papers, each annotated with human-labeled keywords and accompanied by original text, images (with OCR-extracted text), and audio transcripts (via automatic speech recognition). Leveraging this resource, the paper systematically evaluates the impact of individual modalities and their fusion—under both unsupervised and supervised settings—on keyword extraction performance. Experimental results demonstrate that integrating multimodal textual information significantly enhances extraction accuracy, with distinct modalities offering complementary cues, thereby confirming the effectiveness and necessity of multimodal modeling for academic keyword extraction.

Abstract

Up to this point, keyword extraction task typically relies solely on textual data. Neglecting visual details and audio features from image and audio modalities leads to deficiencies in information richness and overlooks potential correlations, thereby constraining the model's ability to learn representations of the data and the accuracy of model predictions. Furthermore, the currently available multimodal datasets for keyword extraction task are particularly scarce, further hindering the progress of research on multimodal keyword extraction task. Therefore, this study constructs a multimodal dataset of academic paper consisting of 1000 samples, with each sample containing paper text, images, audios and keywords. Based on unsupervised and supervised methods of keyword extraction, experiments are conducted using textual data from papers, as well as text extracted from images and audio. The aim is to investigate the differences in performance in keyword extraction task with respect to different modal information and the fusion of multimodal information. The experimental results indicate that text from different modalities exhibits distinct characteristics in the model. The concatenation of paper text, image text and audio text can effectively enhance the keyword extraction performance of academic papers.

Problem

• keyword extraction
• multimodal dataset
• academic paper
• information richness
• cross-modal correlation

Innovation

• multimodal dataset
• keyword extraction
• academic paper
• information fusion
• cross-modal representation