DenseStep2M: A Scalable, Training-Free Pipeline for Dense Instructional Video Annotation

2026-04-29

AI Summary

This work addresses the scarcity of high-quality procedural annotations in long videos caused by ASR noise and audio-visual temporal misalignment. It proposes the first training-free automated pipeline that leverages shot segmentation and cross-modal alignment filtering, followed by structured, temporally aligned fine-grained procedural step generation using multimodal large language models such as Qwen2.5-VL and DeepSeek-R1. This approach enables large-scale, training-free dense procedural video annotation, resulting in DenseStep2M—a novel dataset comprising 100,000 videos and 2 million steps. The method significantly advances performance on dense captioning, step localization, and cross-modal retrieval tasks, while demonstrating strong zero-shot generalization capabilities.

Abstract

Long-term video understanding requires interpreting complex temporal events and reasoning over procedural activities. While instructional video corpora, like HowTo100M, offer rich resources for model training, they present significant challenges, including noisy ASR transcripts and inconsistent temporal alignments between narration and visual content. In this work, we introduce an automated, training-free pipeline to extract high-quality procedural annotations from in-the-wild instructional videos. Our approach segments videos into coherent shots, filters poorly aligned content, and leverages state-of-the-art multimodal and large language models (Qwen2.5-VL and DeepSeek-R1) to generate structured, temporally grounded procedural steps. This pipeline yields DenseStep2M, a large-scale dataset comprising approximately 100K videos and 2M detailed instructional steps, designed to support comprehensive long-form video understanding. To rigorously evaluate our pipeline, we curate DenseCaption100, a benchmark of high-quality, human-written captions. Evaluations demonstrate strong alignment between our auto-generated steps and human annotations. Furthermore, we validate the utility of DenseStep2M across three core downstream tasks: dense video captioning, procedural step grounding, and cross-modal retrieval. Models fine-tuned on DenseStep2M achieve substantial gains in captioning quality and temporal localization, while exhibiting robust zero-shot generalization across egocentric, exocentric, and mixed-perspective domains. These results underscore the effectiveness of DenseStep2M in facilitating advanced multimodal alignment and long-term activity reasoning. Our dataset is available at https://huggingface.co/datasets/mingjige/DenseStep2M.

Problem

• instructional video
• dense annotation
• temporal alignment
• procedural understanding
• video captioning

Innovation

• training-free pipeline
• dense procedural annotation
• multimodal large language models
• temporal grounding
• long-form video understanding