HMCF: A Human-in-the-loop Multi-Robot Collaboration Framework Based on Large Language Models

2025-05-01

AI Summary

To address poor generalizability, low safety, and task-adaptation difficulty of large-scale heterogeneous multi-robot systems in complex scenarios such as disaster response, this paper proposes the first LLM-driven edge-agent collaboration framework. The framework integrates task semantic parsing, capability-aware instruction generation, human-robot collaborative decision-making, and heterogeneous interface abstraction, augmented by a human-in-the-loop intervention mechanism and a task verification feedback loop—enabling zero-shot cross-task and cross-environment deployment. Experiments demonstrate a 4.76% improvement in simulation task success rate and significantly reduced human intervention frequency in real-world deployments, while maintaining high-reliability coordination. The core innovation lies in deeply embedding large language models into the on-device robotic agent architecture, thereby overcoming the strong scene- and task-specific dependencies inherent in conventional approaches without compromising safety—establishing a novel paradigm for general-purpose, human-robot collaborative multi-robot systems.

Technology Category

• Intelligent Robots: Multi-Robot Systems
• Multiagent Systems: Adversarial Agents
• Natural Language Processing: Safety and Robustness

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Rapid advancements in artificial intelligence (AI) have enabled robots to performcomplex tasks autonomously with increasing precision. However, multi-robot systems (MRSs) face challenges in generalization, heterogeneity, and safety, especially when scaling to large-scale deployments like disaster response. Traditional approaches often lack generalization, requiring extensive engineering for new tasks and scenarios, and struggle with managing diverse robots. To overcome these limitations, we propose a Human-in-the-loop Multi-Robot Collaboration Framework (HMCF) powered by large language models (LLMs). LLMs enhance adaptability by reasoning over diverse tasks and robot capabilities, while human oversight ensures safety and reliability, intervening only when necessary. Our framework seamlessly integrates human oversight, LLM agents, and heterogeneous robots to optimize task allocation and execution. Each robot is equipped with an LLM agent capable of understanding its capabilities, converting tasks into executable instructions, and reducing hallucinations through task verification and human supervision. Simulation results show that our framework outperforms state-of-the-art task planning methods, achieving higher task success rates with an improvement of 4.76%. Real-world tests demonstrate its robust zero-shot generalization feature and ability to handle diverse tasks and environments with minimal human intervention.

Problem

• Addressing generalization challenges in multi-robot systems
• Enhancing adaptability and safety with human oversight
• Optimizing task allocation for heterogeneous robot teams

Innovation

• Human-in-the-loop multi-robot collaboration framework
• LLM agents for task understanding and execution
• Task verification and human supervision reduce hallucinations