HEART: Coordination of Heterogeneous Expert Agents for Physically Grounded Robotic Task Planning

2026-06-24

AI Summary

This work addresses the challenge that large language models (LLMs) often generate infeasible or incomplete action sequences in robotic task planning due to neglecting physical and spatial constraints. To overcome this limitation, the authors propose a role-specialized heterogeneous multi-agent framework that decomposes complex instructions into atomic reasoning tasks, which are collaboratively executed by dedicated expert agents. A constraint-driven plan synthesis mechanism ensures the physical feasibility of the resulting plans. The approach introduces, for the first time, a role-based collaboration scheme among heterogeneous LLMs, enabling efficient planning under limited computational and communication resources. Experimental results on multiple household scenario benchmarks demonstrate significantly higher planning success rates compared to both single-LLM and rule-based baselines, confirming the method’s robustness and scalability.

Abstract

Large Language Models (LLMs) can reason over complex instructions but often fail to satisfy the physical and spatial constraints required for robotic task planning. Recent LLM-based planners directly translate text into action sequences, yet they lack structured reasoning about feasibility, reachability, and logical order, resulting in invalid or incomplete plans. We present a heterogeneous multi-LLM framework that decomposes instructions into atomic reasoning tasks and allocates them to role-specialized expert agents under a token budget for real-world computational and communicational constraints. By combining role-oriented reasoning from heterogeneous agents followed by constraint-driven plan synthesis, HEART validates capability, reachability, and constraint conditions before planning and helps produce physically executable plans while maintaining efficiency. Experiments across different household benchmarks show that HEART consistently improves plan success compared to single-LLM and rule-based planners, demonstrating that heterogeneous LLM collaboration enables robust and scalable robotic task planning under resource constraints.

Problem

• robotic task planning
• physical constraints
• large language models
• heterogeneous agents
• plan feasibility

Innovation

• heterogeneous multi-LLM
• robotic task planning
• constraint-driven synthesis
• role-specialized agents
• physically grounded reasoning