MS4UI: A Dataset for Multi-modal Summarization of User Interface Instructional Videos

2025-06-14

AI Summary

Existing general-purpose video summarization benchmarks lack support for stepwise, executable instructions and multimodal alignment between keyframes and textual descriptions in UI tutorial videos. Method: We introduce UI-Teach, the first multimodal summarization benchmark specifically designed for UI tutorial videos, comprising 2,413 videos (167 hours) with human-annotated fine-grained video segmentation, stepwise executable instructions, and corresponding keyframes. We formally define the UI tutorial video summarization task, emphasizing instruction executability and cross-modal consistency, and establish an end-to-end aligned multimodal annotation framework. Contribution/Results: Experiments reveal a substantial performance drop (−28.6% average ROUGE-L) for current state-of-the-art multimodal summarization models on UI-Teach, confirming the task’s inherent difficulty and the need for novel methodological advances. UI-Teach serves as critical infrastructure for skill-oriented video understanding and generation research.

Technology Category

• Computer Vision: Video Understanding & Activity Analysis
• Machine Learning: Multimodal Learning
• Natural Language Processing: Summarization

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

We study multi-modal summarization for instructional videos, whose goal is to provide users an efficient way to learn skills in the form of text instructions and key video frames. We observe that existing benchmarks focus on generic semantic-level video summarization, and are not suitable for providing step-by-step executable instructions and illustrations, both of which are crucial for instructional videos. We propose a novel benchmark for user interface (UI) instructional video summarization to fill the gap. We collect a dataset of 2,413 UI instructional videos, which spans over 167 hours. These videos are manually annotated for video segmentation, text summarization, and video summarization, which enable the comprehensive evaluations for concise and executable video summarization. We conduct extensive experiments on our collected MS4UI dataset, which suggest that state-of-the-art multi-modal summarization methods struggle on UI video summarization, and highlight the importance of new methods for UI instructional video summarization.

Problem

• Lack of benchmarks for UI instructional video summarization
• Need step-by-step executable instructions and illustrations
• Existing methods struggle with concise UI video summarization

Innovation

• Multi-modal summarization for instructional videos
• Novel benchmark for UI video summarization
• Dataset with manual annotations for evaluations