SLVMBench: Skill Learning from Video Memory

2026-07-13

AI Summary

Current video-language foundation models struggle to learn from long-duration videos and transfer acquired skills in real time. This work proposes the first unified evaluation benchmark that simulates how humans extract embedded procedural knowledge from hours-long video streams and immediately apply it to novel tasks. The benchmark innovatively integrates long-term video memory, procedural knowledge extraction, and real-time skill application, featuring sub-second temporal annotations and carefully designed questions resistant to commonsense-based guessing. Its evaluation suite is constructed through long-context comprehension tasks, human-annotated question-answer pairs, and synthetically composited multi-source videos. Comprehensive assessments of leading open- and closed-source models reveal substantial deficiencies in long-video skill acquisition and transfer, highlighting critical limitations in current architectures.

Abstract

We introduce Skill Learning from Video Memory (SLVMBench), the first benchmark that jointly evaluates whether video large language models (video-LLMs) can learn skills from long video memory and apply them to real-time tasks. SLVMBench presents models with 2-3 hour video streams that contain a tutorial video embedded in a stream of arbitrary irrelevant videos, resembling real-world human learning practices. Video-LLMs are asked to apply the acquired skill to answer real-time questions about an ongoing video. Unlike long-video understanding benchmarks that emphasize passive comprehension and skill-learning benchmarks that rely on short, immediate demonstrations, SLVMBench tests the full pipeline of memorizing and extracting procedural knowledge, as well as transferring it to real-time tasks. Moreover, rigorous human annotations feature sub-second-level temporal calibration, manually engineered questions eliminating common-sense guessing, and collated tutorials to ensure coverage of the required skills. Evaluations on state-of-the-art proprietary and open-source video LLMs show that video-LLMs struggle substantially with learning and applying skill knowledge from videos. Moreover, performance degrades markedly when the skill knowledge is placed within a long video memory. These results reveal a key limitation of existing video LLMs and position SLVMBench as the first benchmark for studying real-time skill acquisition and application from long-context video memory.

Problem

• video-LLMs
• skill learning
• long video memory
• real-time tasks
• procedural knowledge

Innovation

• video-LLMs
• skill learning
• long-video memory
• real-time application
• procedural knowledge transfer