PlanLLM: Video Procedure Planning with Refinable Large Language Models

2024-12-26

AI Summary

Existing video procedural planning methods rely on LLMs to generate fixed action sequences, suffering from poor generalization and limited adaptability to novel tasks or open-vocabulary actions. This work addresses video program planning for embodied intelligence—i.e., inferring executable action sequences from given start and goal frames. We propose an LLM-enhanced planning framework featuring: (1) a novel cross-modal joint learning mechanism that maximizes mutual information to bridge world knowledge and instance-level visual semantics; (2) free-form, open-vocabulary action generation; and (3) co-training of action decoding and textual reasoning. Evaluated on three benchmarks, our method achieves state-of-the-art performance, simultaneously attaining high closed-set accuracy and strong open-vocabulary robustness. It significantly improves planning accuracy, flexibility, and generalization across diverse tasks and unseen action vocabularies.

Technology Category

• Planning, Routing, and Scheduling: Planning with Language Models
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Video procedure planning, i.e., planning a sequence of action steps given the video frames of start and goal states, is an essential ability for embodied AI. Recent works utilize Large Language Models (LLMs) to generate enriched action step description texts to guide action step decoding. Although LLMs are introduced, these methods decode the action steps into a closed-set of one-hot vectors, limiting the model's capability of generalizing to new steps or tasks. Additionally, fixed action step descriptions based on world-level commonsense may contain noise in specific instances of visual states. In this paper, we propose PlanLLM, a cross-modal joint learning framework with LLMs for video procedure planning. We propose an LLM-Enhanced Planning module which fully uses the generalization ability of LLMs to produce free-form planning output and to enhance action step decoding. We also propose Mutual Information Maximization module to connect world-level commonsense of step descriptions and sample-specific information of visual states, enabling LLMs to employ the reasoning ability to generate step sequences. With the assistance of LLMs, our method can both closed-set and open vocabulary procedure planning tasks. Our PlanLLM achieves superior performance on three benchmarks, demonstrating the effectiveness of our designs.

Problem

• Video Process Planning
• Large Language Models (LLMs)
• Adaptability to New Situations

Innovation

• PlanLLM
• Video Step Planning
• Large Language Models