NUTSHELL: A Dataset for Abstract Generation from Scientific Talks

2025-02-24

AI Summary

The scientific abstract generation (SAG) task has been hindered by the scarcity of large-scale, high-quality annotated multimodal data. To address this, we introduce NUTSHELL—the first academic-domain multimodal benchmark for SAG—comprising audio recordings, ASR transcripts, and corresponding paper abstracts from ACL conference talks. We propose the first realistic, academically aligned SAG evaluation framework, integrating automated metrics (ROUGE, BERTScore) with human assessment. Leveraging ASR transcripts, we develop two complementary baseline approaches: a Seq2Seq model and an LLM fine-tuning pipeline, both supporting multi-stage summarization and quality analysis. Experimental results demonstrate that our methods significantly outperform prior work across all metrics. NUTSHELL establishes a standardized, reproducible foundation for future SAG research, enabling systematic advancement in multimodal scientific summarization.

Technology Category

• Natural Language Processing: Summarization
• Machine Learning: Large Multimodal Models (LMMs)
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Scientific communication is receiving increasing attention in natural language processing, especially to help researches access, summarize, and generate content. One emerging application in this area is Speech-to-Abstract Generation (SAG), which aims to automatically generate abstracts from recorded scientific presentations. SAG enables researchers to efficiently engage with conference talks, but progress has been limited by a lack of large-scale datasets. To address this gap, we introduce NUTSHELL, a novel multimodal dataset of *ACL conference talks paired with their corresponding abstracts. We establish strong baselines for SAG and evaluate the quality of generated abstracts using both automatic metrics and human judgments. Our results highlight the challenges of SAG and demonstrate the benefits of training on NUTSHELL. By releasing NUTSHELL under an open license (CC-BY 4.0), we aim to advance research in SAG and foster the development of improved models and evaluation methods.

Problem

• Lack of large-scale datasets for Speech-to-Abstract Generation.
• Introducing NUTSHELL dataset to address this gap.
• Evaluating SAG models using NUTSHELL for improved performance.

Innovation

• Novel multimodal dataset introduced
• Strong baselines for SAG established
• Open license CC-BY 4.0 applied