Contrastive Energy Fields for Inference-Time Procedure Planning in Instructional Videos

2026-08-17

AI Summary

This study addresses the limitations of existing process planning methods, which often lack task logic constraints and struggle to exclude irrelevant actions. To overcome these challenges, we propose CEFITO, a novel approach that learns an action-conditioned representation space to reframe process planning as a test-time task-constrained optimization problem. By leveraging contrastive energy fields, CEFITO explicitly eliminates irrelevant actions, ensuring generated sequences adhere to logical constraints. Experimental results demonstrate that CEFITO achieves state-of-the-art accuracy on two mainstream benchmarks, significantly enhancing both logical consistency and dynamic adaptability in planning. This work establishes a new paradigm for complex task planning by integrating explicit constraint satisfaction into the generation process.

Abstract

Procedure planning seeks to estimate a sequence of actions to transition from an observed initial state to a given goal state. Current procedure planning approaches directly predict action sequences from latent representations using feed-forward neural networks or diffusion-based inference. These paradigms treat every action as plausible, lacking the ability to enforce task-specific logical constraints that render certain actions irrelevant or not plausible. We propose CEFITO, a procedure planning approach that learns a predictor to express an action-conditioned representation space. Based on this representation space, we formulate procedure planning as a task-constrained optimization problem. Unlike prior methods, CEFITO explicitly reasons over the action space by omitting irrelevant actions during inference-time planning. This reformulation enables effective procedure planning and achieves state-of-the-art accuracy on two established procedure planning benchmarks.

Problem

• Procedure Planning
• Instructional Videos
• Logical Constraints
• Action Space Reasoning

Innovation

• Contrastive Energy Fields
• Procedure Planning
• Task-Constrained Optimization
• Action-Conditioned Representation
• Inference-Time Planning