FlexLAM: Resolving the Bottleneck Trade-off in Latent Action Learning

2026-06-17

AI Summary

Existing latent action models (LAMs) struggle to balance information retention and compression efficiency under label scarcity or narrow action distributions due to fixed-capacity bottlenecks. This work proposes FlexLAM, which leverages variable-length latent actions and a nested dropout training strategy to enable prefix-efficient encoding. Without modifying the model architecture or introducing auxiliary loss functions, FlexLAM dynamically adjusts action granularity at inference time. It is the first approach to allow a single model to flexibly adapt to arbitrary token budgets, significantly outperforming multiple specialized fixed-capacity LAMs. FlexLAM achieves superior performance in both standard label-scarce supervision and low-reward alignment tasks, and improves action transition reconstruction on Ego4D.

Abstract

Latent actions provide a compact interface between action-free video and downstream decision-making, yet existing Latent Action Models (LAMs) force every transition through a fixed-capacity bottleneck. We identify a bottleneck trade-off: overly tight codes can discard transition cues needed for action alignment, while overly loose codes preserve additional transition variation that must be resolved when alignment labels are scarce or narrowly distributed. FlexLAM replaces this fixed capacity with variable-length latent actions trained by nested dropout, yielding prefix-valid codes that capture compact transition structure first and add detail only when needed, without new architectures or losses. A single FlexLAM matches or surpasses separately trained fixed-capacity LAMs at every evaluated token budget under standard scarce-label supervision and under a low-return single-task alignment stress test, indicating that FlexLAM is not merely adjustable at inference time but learns a better latent-action interface at the same token budgets. The same model supports inference-time token-budget adjustment without retraining, and FlexLAM improves Ego4D transition reconstruction. These results suggest that variable-length latent actions are an architecture-free, drop-in upgrade to the fixed-capacity bottleneck in latent action models, latent-action world models, and video-pretrained action interfaces.

Problem

• Latent Action Models
• bottleneck trade-off
• variable-length latent actions
• action alignment
• token budget

Innovation

• variable-length latent actions
• nested dropout
• bottleneck trade-off
• prefix-valid codes
• latent action models