Generalist Forecasting with Frozen Video Models via Latent Diffusion

2025-07-18

AI Summary

This work addresses the challenge of enhancing general-purpose vision models’ capability for future frame prediction across multiple abstraction levels. We propose a generic prediction framework comprising a frozen backbone and latent diffusion: a pre-trained video model serves as a fixed feature extractor, while a lightweight latent diffusion model operates in its latent space to forecast temporal features; task-specific decoders then generate downstream outputs. We further establish, for the first time, a strong correlation between visual representation quality and short-term prediction performance, and introduce distributional metrics defined in downstream task spaces to directly evaluate prediction fidelity. Extensive experiments across nine models and four diverse video understanding tasks demonstrate significant improvements in short-horizon, multi-level prediction accuracy. Our approach bridges representation learning and generative modeling, advancing their synergistic integration in video understanding.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Large pretrained models with web data
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Forecasting what will happen next is a critical skill for general-purpose systems that plan or act in the world at different levels of abstraction. In this paper, we identify a strong correlation between a vision model's perceptual ability and its generalist forecasting performance over short time horizons. This trend holds across a diverse set of pretrained models-including those trained generatively-and across multiple levels of abstraction, from raw pixels to depth, point tracks, and object motion. The result is made possible by a novel generalist forecasting framework that operates on any frozen vision backbone: we train latent diffusion models to forecast future features in the frozen representation space, which are then decoded via lightweight, task-specific readouts. To enable consistent evaluation across tasks, we introduce distributional metrics that compare distributional properties directly in the space of downstream tasks and apply this framework to nine models and four tasks. Our results highlight the value of bridging representation learning and generative modeling for temporally grounded video understanding.

Problem

• Correlation between vision model perception and forecasting performance
• Generalist forecasting framework using frozen vision backbone
• Bridging representation learning and generative modeling for video understanding

Innovation

• Uses frozen vision models for forecasting
• Trains latent diffusion models for future features
• Introduces distributional metrics for evaluation