Learning Lifted Action Models from Unsupervised Visual Traces

2026-04-20

AI Summary

This work addresses the challenge of automatically learning lifted symbolic action models from visual state sequences without explicit action annotations to support real-world AI planning. The authors propose a joint learning framework that simultaneously optimizes state prediction, action prediction, and the action model itself. For the first time, they integrate mixed-integer linear programming (MILP) to enforce logical consistency constraints on predictions and combine it with a pseudo-labeling self-training mechanism to mitigate error propagation and prevent prediction collapse. Experimental results demonstrate that the approach effectively escapes local optima and converges to globally consistent and accurate action models, with ablation studies across multiple domains highlighting the critical role of MILP-based correction in achieving this performance.

Technology Category

• Humans and AI: Human-Aware Planning and Behavior Prediction
• Planning, Routing, and Scheduling: Learning for Planning and Scheduling
• Machine Learning: Neuro-Symbolic Learning

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Efficient construction of models capturing the preconditions and effects of actions is essential for applying AI planning in real-world domains. Extensive prior work has explored learning such models from high-level descriptions of state and/or action sequences. In this paper, we tackle a more challenging setting: learning lifted action models from sequences of state images, without action observation. We propose a deep learning framework that jointly learns state prediction, action prediction, and a lifted action model. We also introduce a mixed-integer linear program (MILP) to prevent prediction collapse and self-reinforcing errors among predictions. The MILP takes the predicted states, actions, and action model over a subset of traces and solves for logically consistent states, actions, and action model that are as close as possible to the original predictions. Pseudo-labels extracted from the MILP solution are then used to guide further training. Experiments across multiple domains show that integrating MILP-based correction helps the model escape local optima and converge toward globally consistent solutions.

Problem

• lifted action models
• unsupervised learning
• visual traces
• action model learning
• state images

Innovation

• lifted action models
• unsupervised visual traces
• mixed-integer linear programming
• pseudo-labeling
• joint learning