Integrated Sensing and Communications for Real-time Avatar Control in XR over 5G

2026-06-22

AI Summary

Existing XR interaction paradigms struggle to simultaneously achieve unobstructed operation, full-body sensing, and free hand manipulation. This work proposes a multimodal perception framework that integrates 5G millimeter-wave integrated sensing and communication (ISAC) with surface electromyography (sEMG) signals, pioneering the use of communication signals for whole-body pose estimation while leveraging lightweight sEMG for fine-grained finger gesture recognition—eliminating the need for additional dedicated sensors. By employing a power-per-beam-pair (PPBP) sensing approach and a multimodal fusion strategy, the method achieves a body pose recognition accuracy of 82.2 ± 5.9% on unseen users, while sEMG demonstrates strong discriminability and stability across diverse gestures, significantly enhancing the real-time performance and precision of virtual avatar control.

Abstract

Extended Reality (XR) presents a challenging use case for 5G and 6G networks, requiring high data-rates and lowlatency communication to deliver a truly immersive experience. Moreover, in order to seamlessly translate physical actions to the virtual world, accurate gesture recognition and pose estimation are required. Current XR interaction solutions based on handheld controllers and cameras cannot easily capture full-body poses, inhibit the free use of hands, and require good visibility and a clear line of sight. In this work, we propose a multimodal sensing architecture for XR that combines 5G MillimeterWave (mmWave) Integrated sensing and communication (ISAC) and surface electromyography (sEMG) signals. 5G mmWave ISAC cannot only be used to deliver content wirelessly to the Head-mounted display (HMD), but also the same communication signals can be used to derive coarse body-level gestures and poses of the user, to support real-time avatar control. For fine-grained finger-level gestures, our architecture leverages lightweight sEMG sensors that capture forearm muscle activity. To illustrate the need of both modalities, we present evaluations of both sensing technologies. At the body level (5G), our architecture relies on power-per-beam-pair (PPBP), which can be computed from standard beam management or beam sweeping procedures of the 5G NR standard. PPBP-based sensing achieves 82.2$\pm$5.9% average accuracy when evaluated on users not seen during training. For fine-grained finger-level interactions, we show that surface electromyography (sEMG) carries strong discriminative information achieving consistent promising performance across different movement settings. Thus, combining the two modalities enables multi-scale gesture recognition, at the body level via existing 5G signals and finger level via lightweight sEMG sensors, forming a complete XR framework.

Problem

• Extended Reality
• gesture recognition
• pose estimation
• Integrated Sensing and Communications
• real-time avatar control

Innovation

• Integrated Sensing and Communications (ISAC)
• surface electromyography (sEMG)
• multimodal sensing
• real-time avatar control
• 5G mmWave