SUNTA: Hierarchical Video Prediction with Surprise-based Chunking

2026-07-02

AI Summary

Existing hierarchical state-space models for video prediction struggle to align with the intrinsic temporal structure of data due to their reliance on fixed or similarity-based temporal chunking, which limits long-horizon predictive performance. This work proposes a hierarchical video prediction model that dynamically segments time based on prediction error—interpreted as a surprise signal—thereby introducing, for the first time, a surprise-driven chunking mechanism into this framework. To prevent hierarchical collapse, the model employs a decoupled training strategy and leverages internal inconsistency as a top-down measure of surprise to precisely identify chunk boundaries during open-loop imagination rollouts. Experiments demonstrate that the proposed method significantly outperforms baselines on both 2D and 3D video prediction tasks, being the only model capable of maintaining high prediction accuracy up to 250 steps, whereas existing approaches typically degrade sharply within the first 10 steps.

Abstract

Hierarchical state-space models (HSSMs) offer a promising approach to long-horizon prediction by segmenting sequences into temporal chunks. However, their performance hinges on how chunk boundaries are determined. While prior HSSMs typically rely on fixed-length chunking or similarity-based boundary detection, these methods often misalign with the intrinsic temporal structure of the data. We argue that chunking should instead be driven by prediction errors, which more directly indicate when longer-range context becomes necessary. Nevertheless, integrating surprise-based chunking into HSSMs introduces critical challenges, including hierarchical collapse during end-to-end training and the absence of surprise signals during open-loop prediction. To address these issues, we propose Surprise-based Nested Temporal Abstraction (SUNTA), a method that employs a decoupled training strategy to preserve surprise signals and uses internal inconsistency as a top-down surprise metric to determine chunk boundaries within imagined rollouts. Experiments on video prediction tasks in 2D and 3D environments demonstrate that SUNTA outperforms baselines, uniquely maintaining accurate predictions over 250 timesteps, whereas all baselines degrade within the first 10 timesteps.

Problem

• hierarchical video prediction
• temporal chunking
• surprise-based chunking
• state-space models
• long-horizon prediction

Innovation

• surprise-based chunking
• hierarchical state-space models
• video prediction
• temporal abstraction
• decoupled training