Understanding and Mitigating the Video-Action Generalization Gap via Temporal Ratio

2026-07-09

AI Summary

This work addresses the degradation in compositional generalization when fine-tuning video foundation models for robotic action generation—a phenomenon termed the video-to-action generalization gap. To tackle this issue, the authors propose Temporal Ratio (TR), an attention-based metric that quantifies the degree to which an action head relies on future latent frames. TR effectively predicts a model’s compositional generalization performance and naturally varies across task phases. Leveraging this insight, the authors introduce an inference-time adaptive guidance mechanism that dynamically strengthens video conditioning signals. Evaluated on the LIBERO benchmark and real-world tasks, this approach significantly narrows the generalization gap between in-distribution and out-of-distribution scenarios.

Abstract

Generative video foundation models exhibit strong compositional priors, yet world-action models (WAMs) and video-action models (VAMs) often lose these priors after finetuning on robotic action data. We refer to this discrepancy as the video-action generalization gap. In this paper, we systematically investigate this gap by evaluating a comprehensive design space of VAMs, demonstrating that standard design choices yield no emergent explanation pattern. To explain this behavior, we introduce the Temporal Ratio (TR), an attention-based measure of how strongly the action head relies on future latent rollouts relative to the anchored current frame. TR has two key properties: first, a model's structural reliance on future-predictive latents, measured via TR, acts as a predictor of its compositional generalization capacity; second, it natively fluctuates based on task phase, shifting attention to future frames during planning and reverting to the present frame for precise manipulation. Finally, based on these findings, we propose an inference-time adaptive guidance method, which exploits this intrinsic feature attention pattern to dynamically amplify compositional video conditioning signals precisely when the policy relies on future rollouts. Evaluated on the LIBERO benchmark and real-world tasks, our approach mitigates the OOD-ID compositional generalization gap. More details: https://umishra.me/temporal-ratio/

Problem

• video-action generalization gap
• compositional generalization
• temporal ratio
• foundation models
• robotic action

Innovation

• Temporal Ratio
• video-action generalization gap
• compositional generalization
• adaptive guidance
• foundation models