A Good Talk Does not Look Like a Summary, It Teaches You! Measuring Takeaways from Paper-to-Video Talks

2026-06-26

AI Summary

This work addresses a critical gap in current paper-to-video systems, which typically evaluate only content coverage while neglecting audience comprehension of core scientific ideas. To bridge this gap, the authors propose EffectivePresentationScorer, a novel framework that, for the first time, focuses on the pedagogical effectiveness of scientific presentation videos. By integrating natural language understanding with principles from educational assessment, the framework establishes a multidimensional scoring mechanism to evaluate script quality across key dimensions such as logical structure, contextual framing, and clarity of technical explanations. Experimental results reveal that while automatically generated videos often cover topical content and maintain structural coherence, they consistently fail to clearly articulate prerequisite concepts and methodological rationale. These findings expose a significant blind spot in existing evaluation metrics and establish the first automated approach for assessing the instructional quality of research communication videos.

Abstract

Automatically generated videos from scientific papers are increasingly used for education and research dissemination. However, existing evaluation metrics mainly measure visual quality or whether key points from the paper appear in the video without assessing whether the video actually helps viewers understand the ideas. We introduce EffectivePresentationScorer, a framework for evaluating the instructional quality of scientific presentation videos. It checks whether a video explains the main ideas clearly, introduces needed background concepts, and connects technical details to the main contribution of the paper. When we apply EffectivePresentationScorer to the existing paper-to-video generation systems, we find that generated videos mention the correct topics and follow the structure of the paper but fail to explain prerequisite concepts or clarify why the method works. These failures are often ignored by existing video evaluation metrics, which focus on content presence rather than explanatory quality.

Problem

• scientific video evaluation
• instructional quality
• paper-to-video generation
• explanatory quality
• takeaway understanding

Innovation

• EffectivePresentationScorer
• instructional quality
• paper-to-video generation
• explanatory evaluation
• scientific communication