Bridging Industrial Expertise and XR with LLM-Powered Conversational Agents

2025-04-07

AI Summary

To address the challenge of inefficient knowledge transfer among manufacturing domain experts, this paper proposes a hands-free industrial knowledge assistant integrating Extended Reality (XR) with Retrieval-Augmented Generation (RAG)-enabled large language models (LLMs). Methodologically, it introduces a novel synergistic optimization paradigm comprising industrial semantic chunking, balanced multilingual and multimodal embedding (BGE), and lightweight vector indexing (FAISS/Weaviate), coupled with dynamic LLM tool orchestration and real-time XR interaction. The system—built on Unity/Unreal—features speech-driven ASR/TTS and context-aware on-site guidance. Evaluated in robot assembly, smart infrastructure maintenance, and aero-engine component inspection, it achieves a 40% improvement in knowledge retrieval training efficiency and end-to-end remote collaboration latency under 1.2 seconds, thereby advancing Industry 5.0 objectives of human-centricity and operational resilience.

Technology Category

• Knowledge Representation and Reasoning: Knowledge Engineering
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

This paper introduces a novel integration of Retrieval-Augmented Generation (RAG) enhanced Large Language Models (LLMs) with Extended Reality (XR) technologies to address knowledge transfer challenges in industrial environments. The proposed system embeds domain-specific industrial knowledge into XR environments through a natural language interface, enabling hands-free, context-aware expert guidance for workers. We present the architecture of the proposed system consisting of an LLM Chat Engine with dynamic tool orchestration and an XR application featuring voice-driven interaction. Performance evaluation of various chunking strategies, embedding models, and vector databases reveals that semantic chunking, balanced embedding models, and efficient vector stores deliver optimal performance for industrial knowledge retrieval. The system's potential is demonstrated through early implementation in multiple industrial use cases, including robotic assembly, smart infrastructure maintenance, and aerospace component servicing. Results indicate potential for enhancing training efficiency, remote assistance capabilities, and operational guidance in alignment with Industry 5.0's human-centric and resilient approach to industrial development.

Problem

• Addressing knowledge transfer challenges in industrial environments using XR and LLMs
• Embedding domain-specific industrial knowledge into XR for hands-free expert guidance
• Enhancing training efficiency and remote assistance in Industry 5.0 applications

Innovation

• RAG-enhanced LLMs for industrial knowledge transfer
• XR with voice-driven natural language interface
• Semantic chunking optimizes knowledge retrieval performance