Beyond the Lens: Quantifying the Impact of Scientific Documentaries through Amazon Reviews

2025-02-12

AI Summary

This study addresses the lack of large-scale, quantitative methods for assessing public impact of science documentaries. We propose the first fine-grained impact taxonomy tailored to this domain and construct an open-source dataset comprising 1,296 human-annotated sentences drawn from 1,043 viewer comments on six popular Amazon science documentaries. Our methodology integrates machine learning with large language models to jointly identify impact categories—namely, sparking interest, enhancing understanding, and prompting action—and extract associated scientific themes, augmented by quantitative sentiment analysis and expert validation. Experimental results achieve a maximum F1-score of 0.89 and demonstrate strong cross-documentary generalization. Key contributions include: (1) the first fine-grained evaluation framework for science documentary impact; (2) the first publicly available, manually annotated benchmark dataset; and (3) a scalable hybrid modeling paradigm. This work establishes a new empirical foundation and methodological benchmark for evaluating science communication effectiveness.

Technology Category

• Application Domains: Humanities & Computational Social Science
• Natural Language Processing: Discourse, Pragmatics & Argument Mining
• Machine Learning: Learning Preferences or Rankings

Application Category

• Web Mining and Content Analysis: Sentiment analysis and opinion mining
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

Engaging the public with science is critical for a well-informed population. A popular method of scientific communication is documentaries. Once released, it can be difficult to assess the impact of such works on a large scale, due to the overhead required for in-depth audience feedback studies. In what follows, we overview our complementary approach to qualitative studies through quantitative impact and sentiment analysis of Amazon reviews for several scientific documentaries. In addition to developing a novel impact category taxonomy for this analysis, we release a dataset containing 1296 human-annotated sentences from 1043 Amazon reviews for six movies created in whole or part by a team of visualization designers who focus on cinematic presentations of scientific data. Using this data, we train and evaluate several machine learning and large language models, discussing their effectiveness and possible generalizability for documentaries beyond those focused on for this work. Themes are also extracted from our annotated dataset which, along with our large language model analysis, demonstrate a measure of the ability of scientific documentaries to engage with the public.

Problem

• Quantify impact of scientific documentaries
• Analyze Amazon reviews for sentiment
• Develop taxonomy for documentary impact

Innovation

• quantitative impact analysis
• sentiment analysis techniques
• machine learning models