Point Tracking Improves World Action Models

2026-05-22

AI Summary

Existing pixel-level world models are often sensitive to irrelevant visual variations such as lighting and texture, hindering robust modeling of environmental dynamics. This work proposes JOPAT—a joint pixel-and-point-trajectory world-action model that leverages a single denoising diffusion Transformer to simultaneously predict latent visual observations, visibility-aware 2D point trajectories, and actions. By explicitly incorporating point trajectories to represent motion, JOPAT maintains robustness under challenging conditions including occlusions and partial object out-of-frame scenarios, while effectively capturing long-horizon dynamics. Experimental results demonstrate that JOPAT significantly outperforms purely pixel-based baselines on both LIBERO and real-world LeRobot tasks, with particularly strong performance in long-horizon settings involving occlusion, object interaction, and out-of-frame motion.

Abstract

Robot policy learning benefits from world-action models that capture environment dynamics, but pixel-level prediction entangles dynamics with nuisance factors such as lighting and texture, making learned representations vulnerable to task-irrelevant visual variation. We propose JOPAT, a JOint Pixel-And-Track World-Action Model that predicts latent visual observations, 2D point tracks with visibility, and actions in a single denoising diffusion transformer. The key insight is that tracks provide an explicit representation of motion that captures long-horizon dynamics and remains robust under occlusion or partial out-of-frame motion, offering greater utility than modeling pixel appearance alone. On LIBERO and real-world LeRobot tasks, JOPAT improves over pixel-based baselines, with the largest gains on long-horizon tasks involving occlusion, object interaction, and off-screen motion.

Problem

• world-action models
• pixel-level prediction
• visual variation
• occlusion
• long-horizon dynamics

Innovation

• point tracking
• world-action model
• diffusion transformer
• long-horizon dynamics
• visual representation learning