MistyPilot: An Agentic Fast-Slow Thinking LLM Framework for Misty Social Robots

2026-03-03

AI Summary

This work addresses the challenge non-programming users face in translating high-level instructions into executable tool calls and parameter configurations for social robots, which often leads to task failure and unnatural interactions. To this end, we propose MistyPilot, a novel framework that integrates dual-process (fast-and-slow) reasoning with a two-agent architecture—comprising a Physical Interaction Agent (PIA) and a Social Interaction Agent (SIA)—to enable autonomous tool selection and orchestration, emotionally aligned dialogue generation, and user preference awareness. MistyPilot balances low-latency responsiveness with high task completion rates and introduces the first multidimensional evaluation benchmark for Misty robots. Experimental results demonstrate that MistyPilot significantly outperforms baseline methods across key metrics, including routing accuracy, task completeness, retrieval efficiency, tool extensibility, and emotional alignment. We also release five benchmark datasets and the full source code to support reproducibility and future research.

Technology Category

• Intelligent Robots: Human-Robot Interaction
• Natural Language Processing: Safety and Robustness
• Humans and AI: Interaction Techniques and Devices

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Agentic search
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

With the availability of open APIs in social robots, it has become easier to customize general-purpose tools to meet users' needs. However, interpreting high-level user instructions, selecting and configuring appropriate tools, and executing them reliably remain challenging for users without programming experience. To address these challenges, we introduce MistyPilot, an agentic LLM-driven framework for autonomous tool selection, orchestration, and parameter configuration. MistyPilot comprises two core components: a Physically Interactive Agent (PIA) and a Socially Intelligent Agent (SIA). The PIA enables robust sensor-triggered and tool-driven task execution, while the SIA generates socially intelligent and emotionally aligned dialogue. MistyPilot further integrates a fast-slow thinking paradigm to capture user preferences, reduce latency, and improve task efficiency. To comprehensively evaluate MistyPilot, we contribute five benchmark datasets. Extensive experiments demonstrate the effectiveness of our framework in routing correctness, task completeness, fast-slow thinking retrieval efficiency, tool scalability,and emotion alignment. All code, datasets, and experimental videos will be made publicly available on the project webpage.

Problem

• social robots
• tool selection
• user instructions
• autonomous execution
• non-programming users

Innovation

• Agentic LLM
• Fast-Slow Thinking
• Social Robot
• Tool Orchestration
• Emotion Alignment