Beyond Static Instruction: A Multi-agent AI Framework for Adaptive Augmented Reality Robot Training

2026-01-31

HRI Companion

AI Summary

针对AR机器人训练中静态指导无法适配学习者认知差异的问题，提出基于多智能体AI框架的方法，利用LLM智能体实时融合多模态输入，动态调整AR教学内容。

Technology Category

• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Machine Learning: Adversarial Learning & Robustness
• Multiagent Systems: Multiagent Learning

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Augmented Reality (AR) offers powerful visualization capabilities for industrial robot training, yet current interfaces remain predominantly static, failing to account for learners' diverse cognitive profiles. In this paper, we present an AR application for robot training and propose a multi-agent AI framework for future integration that bridges the gap between static visualization and pedagogical intelligence. We report on the evaluation of the baseline AR interface with 36 participants performing a robotic pick-and-place task. While overall usability was high, notable disparities in task duration and learner characteristics highlighted the necessity for dynamic adaptation. To address this, we propose a multi-agent framework that orchestrates multiple components to perform complex preprocessing of multimodal inputs (e.g., voice, physiology, robot data) and adapt the AR application to the learner's needs. By utilizing autonomous Large Language Model (LLM) agents, the proposed system would dynamically adapt the learning environment based on advanced LLM reasoning in real-time.

Problem

• Augmented Reality
• robot training
• adaptive learning
• cognitive profiles
• static interface

Innovation

• multi-agent AI
• adaptive augmented reality
• large language model (LLM)
• multimodal input
• personalized robot training