BFMTrack: Latent Sequence Optimization for Physics-Based Motion Tracking with Behavioral Foundation Models

2026-06-23

AI Summary

Existing behavioral foundation models (BFMs) struggle to accurately track time-varying targets—such as complex motion sequences—due to the absence of temporal dynamics in their latent spaces. This work proposes Latent Sequence Optimization (LSO), a method that directly optimizes temporally coherent latent trajectories within the BFM latent space. By integrating physics-based simulation rollouts with policy gradient updates, LSO achieves precise motion tracking without requiring handcrafted reward functions. The approach innovatively incorporates temporally correlated noise modeling, substantially enhancing trajectory smoothness and detail fidelity, thereby overcoming the limitation of BFMs to time-invariant tasks. Experiments demonstrate that LSO enables high-fidelity, highly generalizable motion reproduction across diverse scenarios, including dense trajectory tracking, sparse keyframe control, and real-world deployment on humanoid robots.

Abstract

Behavioral Foundation Models (BFMs) offer a promising path toward universal physics-based character control by organizing a rich repertoire of physically plausible behaviors into a latent space, guided by a large-scale motion dataset. While these models excel at time-invariant tasks, such as goal-reaching and state-based reward optimization, their latent space does not directly support time-varying objectives, such as tracking a motion sequence. For tracking, existing heuristics rely on moving-window-averaging that fails to capture the nuances of highly dynamic motions. In this work, we propose a novel Latent Sequence Optimization (LSO) to address these shortcomings. Our approach combines simulation rollouts with a policy gradient update to optimize over a sequence of latents, extending the capabilities of BFMs toward precise motion tracking without requiring reward engineering and tuning. To guide the optimization toward smooth, coherent latent trajectories, we model the latent sequence using temporally correlated noise. We validate our approach across dense tracking, sparse keyframing, and direct deployment onto a real humanoid robot.

Problem

• motion tracking
• Behavioral Foundation Models
• time-varying objectives
• latent space
• dynamic motion

Innovation

• Latent Sequence Optimization
• Behavioral Foundation Models
• Physics-Based Motion Tracking
• Policy Gradient
• Temporal Coherence