Problem
Research questions and friction points this paper is trying to address.
Latent Action Models
transition ambiguity
multi-object scenes
distractor-rich environments
action source ambiguity
Latent Actions from Factorized Transition Effects under Agent Ambiguity
🤖 AI Summary
This work addresses the challenge of ambiguous action attribution in unsupervised settings, where observed transitions in multi-object, clutter-rich environments are confounded by agent actions, distractors, camera motion, and background dynamics. To resolve this ambiguity, the paper proposes Observation Transition Factorization (OTF), a method that decomposes transitions into sparse primitives to construct an intermediate representation capable of extracting robust latent action variables. OTF integrates inverse and forward dynamics modeling and operates within a frozen DINOv2 visual representation space, enabling decoder-free future state prediction. The approach demonstrates strong primitive reusability and zero-shot transfer capabilities, achieving competitive or superior performance over existing baselines in downstream policy learning under complex transition ambiguities.
📝 Abstract
Latent Action Models (LAMs) learn action-like proxies from observation transitions. However, in multi-object or distractor-rich scenes, these visual effects mix agent motion with distractors, camera dynamics, and background changes, making the underlying action source ambiguous without supervision. Structuring this mixture as reusable transition effects provides an intermediate representation from which action-like latents can be more robustly formed. We introduce Observed Transition Factorization (OTF), which decomposes each transition into a sparse set of observed transition primitives. Using these primitives as the transition interface, we propose OTF-LAM, which abstracts motion primitives into action-like latents within the standard inverse-forward dynamics framework, and OTF-LAM-Dino, a decoder-free variant that predicts future states in a frozen DINOv2 representation space. Empirically, OTF primitives transfer zeroshot across controlled carrier and morphology shifts, showing reusability. Furthermore, downstream policy learning results match or outperform baselines under complex transition ambiguity.
Innovation
Methods, ideas, or system contributions that make the work stand out.
Latent Action Models
Transition Factorization
Distractor-Robust Representation
Zero-shot Transfer
DINOv2
🔎 Similar Papers
No similar papers found.
💼 Related Jobs
