Hand by Hand: LLM Driving EMS Assistant for Operational Skill Learning

2025-08-08

AI Summary

Current LLM-assisted skill acquisition heavily relies on textual feedback, neglecting kinesthetic perception—a critical sensory channel for procedural tasks. To address this, we propose FlightAxis, the first system integrating large language models (LLMs) with electro-muscular stimulation (EMS) to establish an “align–analyze–adjust” closed-loop framework that enables end-to-end automation from textual operation manuals to real-time limb guidance. Evaluated in flight simulation training, FlightAxis significantly reduces task completion time (p < 0.01), enhances users’ awareness of operational errors and engagement, achieves high user acceptance, and induces neither motor dependency nor cognitive overload. This work pioneers an LLM-driven embodied kinesthetic feedback paradigm, offering a scalable, minimally invasive, multimodal human–AI collaboration pathway for procedural skill learning.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Humans and AI: Human-in-the-loop Machine Learning

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

Operational skill learning, inherently physical and reliant on hands-on practice and kinesthetic feedback, has yet to be effectively replicated in large language model (LLM)-supported training. Current LLM training assistants primarily generate customized textual feedback, neglecting the crucial kinesthetic modality. This gap derives from the textual and uncertain nature of LLMs, compounded by concerns on user acceptance of LLM driven body control. To bridge this gap and realize the potential of collaborative human-LLM action, this work explores human experience of LLM driven kinesthetic assistance. Specifically, we introduced an "Align-Analyze-Adjust" strategy and developed FlightAxis, a tool that integrates LLM with Electrical Muscle Stimulation (EMS) for flight skill acquisition, a representative operational skill domain. FlightAxis learns flight skills from manuals and guides forearm movements during simulated flight tasks. Our results demonstrate high user acceptance of LLM-mediated body control and significantly reduced task completion times. Crucially, trainees reported that this kinesthetic assistance enhanced their awareness of operation flaws and fostered increased engagement in the training process, rather than relieving perceived load. This work demonstrated the potential of kinesthetic LLM training in operational skill acquisition.

Problem

• LLMs lack kinesthetic feedback in operational skill training
• User acceptance of LLM-driven body control is uncertain
• Current LLM assistants neglect physical movement guidance

Innovation

• LLM integrates with EMS for skill training
• Align-Analyze-Adjust strategy enhances learning
• FlightAxis guides forearm movements effectively