WAM-TTT: Steering World-Action Models by Watching Human Play at Test Time

2026-07-08

AI Summary

This work addresses the challenge of adapting frozen robotic foundation models to novel tasks or user preferences without relying on robot demonstrations, task-specific fine-tuning, or human annotations. The authors propose a test-time training framework that leverages self-supervised video prediction to inject unlabeled human demonstration videos into a lightweight adaptive memory module, enabling flexible behavioral guidance. This approach achieves efficient and reusable behavior alignment solely from human videos at test time—without requiring robot actions or manual labels—for the first time. By incorporating meta-training to bridge the gap between human and robot behaviors, the method preserves the base model’s generalization capabilities while significantly outperforming context-based baselines that condition on human videos across diverse manipulation and generalization tasks.

Abstract

Steering robot foundation models (RFMs) toward new task variants or user-preferred behaviors remains challenging, often requiring additional robot demonstrations, task-specific fine-tuning, or long-context conditioning. We present WAM-TTT, a test-time training framework for steering world action models from raw human videos. Rather than treating human videos as trajectories to imitate, WAM-TTT absorbs them into a lightweight adaptive memory inside a frozen WAM through self-supervised video prediction. To make this memory useful for control, we introduce a meta-training stage that aligns human demonstrations with robot behaviors using paired human-robot data and a key--value memory reconstruction objective. At test time, only unlabeled human videos are required to adapt the memory, while the pretrained WAM remains frozen. This enables efficient and reusable steering without robot actions, human-side annotations, or task-specific fine-tuning, while preserving the generalization ability of the foundation model. Extensive experiments show that WAM-TTT consistently outperforms in-context human-video conditioning baselines across diverse manipulation tasks and generalization settings.

Problem

• robot foundation models
• test-time adaptation
• human video guidance
• task generalization
• behavior steering

Innovation

• test-time training
• world-action models
• human video steering
• adaptive memory
• self-supervised video prediction