Action- and Language-Conditioned Video Assessment for Embodied Control

2026-08-08

AI Summary

Existing methods struggle to accurately evaluate the full task trajectories of embodied agents executing multi-step natural language instructions, often overlooking critical intermediate states. To address this limitation, this work proposes ALVA, an evaluator that introduces, for the first time, a joint action–language conditioning framework for video trajectory assessment. Leveraging a pretrained vision–language model, ALVA first summarizes inter-frame visual changes conditioned on the agent’s action sequence and then generates discrete progress scores aligned with the instruction. This approach enables fine-grained, interpretable judgments of intermediate execution stages, achieves near-zero false positives in 3D household simulation environments, and—when used as terminal feedback—significantly enhances policy optimization performance, effectively narrowing the gap to oracle labels.

Abstract

Vision-based embodied agents executing multi-step natural language instructions require feedback mechanisms that assess task progress over complete trajectories. Conventional approaches based on final-frame matching or continuous embedding similarity may overlook intermediate transitions that are necessary for determining whether an instruction has been completed. We propose ALVA (Action- and Language-Conditioned Video Assessment), a trajectory evaluator that conditions its assessment on visual observations, the executed action sequence, and the natural language instruction. The method uses a pre-trained vision-language model (VLM) in two stages: it first summarizes frame-to-frame visual transitions conditioned on the executed actions and then assesses the generated summary with respect to the instruction to produce a discrete trajectory-level progress score. In simulated 3D household environments, ALVA exhibits a conservative assessment pattern with near-zero false-positive rates. When used as terminal feedback for closed-loop policy optimization, it provides more effective feedback than the evaluated static image and embedding-based visual baselines and reduces the performance gap to a ground-truth oracle. These results support action- and language-conditioned video assessment as an interpretable feedback mechanism for the evaluated simulated embodied-control tasks.

Problem

• embodied control
• video assessment
• natural language instructions
• task progress evaluation
• trajectory feedback

Innovation

• action-conditioned assessment
• language-conditioned video evaluation
• vision-language model
• embodied control
• trajectory-level feedback