Graph-of-Constraints Model Predictive Control for Reactive Multi-agent Task and Motion Planning

2026-03-18

AI Summary

Existing approaches to task and motion planning (TAMP) for multi-agent systems struggle to effectively handle partially ordered tasks, dynamic agent assignment, and real-time adaptation under disturbances. To address these challenges, this work proposes GoC-MPC, a novel framework that integrates graph-structured constraint modeling with model predictive control (MPC). Leveraging 3D geometric constraints derived from keypoints and purely visual observations, GoC-MPC enables online replanning without requiring training data or a priori environment models. The method supports dynamic task allocation and execution of partially ordered tasks while maintaining robustness in the presence of perturbations. Experimental results demonstrate that GoC-MPC significantly outperforms baseline methods by improving task success rates, reducing path lengths, and substantially accelerating TAMP computation.

Technology Category

• Multiagent Systems: Multiagent Planning
• Planning, Routing, and Scheduling: Replanning and Plan Repair
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Sequences of interdependent geometric constraints are central to many multi-agent Task and Motion Planning (TAMP) problems. However, existing methods for handling such constraint sequences struggle with partially ordered tasks and dynamic agent assignments. They typically assume static assignments and cannot adapt when disturbances alter task allocations. To overcome these limitations, we introduce Graph-of-Constraints Model Predictive Control (GoC-MPC), a generalized sequence-of-constraints framework integrated with MPC. GoC-MPC naturally supports partially ordered tasks, dynamic agent coordination, and disturbance recovery. By defining constraints over tracked 3D keypoints, our method robustly solves diverse multi-agent manipulation tasks-coordinating agents and adapting online from visual observations alone, without relying on training data or environment models. Experiments demonstrate that GoC-MPC achieves higher success rates, significantly faster TAMP computation, and shorter overall paths compared to recent baselines, establishing it as an efficient and robust solution for multi-agent manipulation under real-world disturbances. Our supplementary video and code can be found at https://sites.google.com/view/goc-mpc/home .

Problem

• multi-agent Task and Motion Planning
• geometric constraints
• partially ordered tasks
• dynamic agent assignment
• disturbance recovery

Innovation

• Graph-of-Constraints
• Model Predictive Control
• Multi-agent TAMP
• Dynamic Agent Coordination
• Visual Keypoint Constraints