APEX: Empowering LLMs with Physics-Based Task Planning for Real-time Insight

2025-05-20

AI Summary

Current large language models (LLMs) lack explicit physical interaction modeling capabilities, limiting their ability to execute dynamic, real-world tasks. To address this, we propose a physics-driven anticipatory task planning framework that establishes a real-time perception–prediction–decision closed loop. Our method introduces structured dynamic interaction graphs—jointly leveraging graph neural networks and differentiable rigid-body dynamics forward simulation—to enable low-latency, differentiable physical state updates and causal action evaluation. A vision-language model (VLM) perception interface synergizes with LLM-based reasoning to form an end-to-end physics-augmented planner. We evaluate on three benchmarks: physical reasoning, long-horizon Tetris planning, and dynamic obstacle avoidance. Our approach significantly outperforms state-of-the-art methods across all tasks, demonstrating that explicit physics modeling is critical for enhancing LLMs’ generalization and robustness in real-world task execution.

Technology Category

• Humans and AI: Human-Aware Planning and Behavior Prediction
• Planning, Routing, and Scheduling: Planning with Language Models
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

Large Language Models (LLMs) demonstrate strong reasoning and task planning capabilities but remain fundamentally limited in physical interaction modeling. Existing approaches integrate perception via Vision-Language Models (VLMs) or adaptive decision-making through Reinforcement Learning (RL), but they fail to capture dynamic object interactions or require task-specific training, limiting their real-world applicability. We introduce APEX (Anticipatory Physics-Enhanced Execution), a framework that equips LLMs with physics-driven foresight for real-time task planning. APEX constructs structured graphs to identify and model the most relevant dynamic interactions in the environment, providing LLMs with explicit physical state updates. Simultaneously, APEX provides low-latency forward simulations of physically feasible actions, allowing LLMs to select optimal strategies based on predictive outcomes rather than static observations. We evaluate APEX on three benchmarks designed to assess perception, prediction, and decision-making: (1) Physics Reasoning Benchmark, testing causal inference and object motion prediction; (2) Tetris, evaluating whether physics-informed prediction enhances decision-making performance in long-horizon planning tasks; (3) Dynamic Obstacle Avoidance, assessing the immediate integration of perception and action feasibility analysis. APEX significantly outperforms standard LLMs and VLM-based models, demonstrating the necessity of explicit physics reasoning for bridging the gap between language-based intelligence and real-world task execution. The source code and experiment setup are publicly available at https://github.com/hwj20/APEX_EXP .

Problem

• Enhancing LLMs with physics-based interaction modeling for real-world tasks
• Addressing limitations of current methods in dynamic object interaction modeling
• Providing low-latency physics simulations for optimal task planning

Innovation

• APEX integrates physics-driven foresight for real-time planning
• Structured graphs model dynamic environmental interactions
• Low-latency simulations enable predictive action selection