SYMBOLIZER: Symbolic Model-free Task Planning with VLMs

2026-04-20

AI Summary

This work proposes the first general-purpose symbolic planning framework that requires neither task-specific engineering nor explicit action models. Addressing the high development cost and limited generalization of traditional task and motion planning (TAMP) approaches—which rely on handcrafted or annotation-derived symbolic models—the method leverages vision-language models (VLMs) for symbol grounding, automatically constructing symbolic states from images using only lifted predicates. Planning is achieved through goal counting combined with a width-based, domain-independent heuristic search. Evaluated on the ProDG and ViPlan benchmarks, the approach achieves state-of-the-art performance, substantially outperforming direct VLM-based planning methods and demonstrating strong zero-shot generalization across diverse domains.

Technology Category

• Computer Vision: Visual Reasoning & Symbolic Representations
• Planning, Routing, and Scheduling: Planning with Language Models
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Traditional Task and Motion Planning (TAMP) systems depend on physics models for motion planning and discrete symbolic models for task planning. Although physics model are often available, symbolic models (consisting of symbolic state interpretation and action models) must be meticulously handcrafted or learned from labeled data. This process is both resource-intensive and constrains the solution to the specific domain, limiting scalability and adaptability. On the other hand, Visual Language Models (VLMs) show desirable zero-shot visual understanding (due to their extensive training on heterogeneous data), but still achieve limited planning capabilities. Therefore, integrating VLMs with classical planning for long-horizon reasoning in TAMP problems offers high potential. Recent works in this direction still lack generality and depend on handcrafted, task-specific solutions, e.g. describing all possible objects in advance, or using symbolic action models. We propose a framework that generalizes well to unseen problem instances. The method requires only lifted predicates describing relations among objects and uses VLMs to ground them from images to obtain the symbolic state. Planning is performed with domain-independent heuristic search using goal-count and width-based heuristics, without need for action models. Symbolic search over VLM-grounded state-space outperforms direct VLM-based planning and performs on par with approaches that use a VLM-derived heuristic. This shows that domain-independent search can effectively solve problems across domains with large combinatorial state spaces. We extensively evaluate on extensively evaluate our method and achieve state-of-the-art results on the ProDG and ViPlan benchmarks.

Problem

• Task and Motion Planning
• Symbolic Models
• Visual Language Models
• Domain Generalization
• Long-horizon Planning

Innovation

• Symbolic Planning
• Visual Language Models
• Model-free Task Planning
• Lifted Predicates
• Domain-independent Search