Hierarchical Reinforcement Learning Framework for Adaptive Walking Control Using General Value Functions of Lower-Limb Sensor Signals

2025-07-22

AI Summary

Addressing the challenges of adaptive locomotion control and insufficient decision robustness for lower-limb exoskeletons operating on variable terrain, this paper proposes a hierarchical reinforcement learning (HRL) framework integrated with General Value Functions (GVFs). The high-level policy leverages multimodal sensor signals—including electromyography, plantar pressure, and joint angles—to enable continuous future-state prediction and temporal abstraction via GVFs; the low-level controller utilizes these predictions to optimize real-time motion execution. Our key contribution lies in embedding GVFs within the HRL architecture, endowing the system with anticipatory perception and terrain classification capability under uncertainty. Experimental results demonstrate significant improvements in classification accuracy across challenging scenarios—such as level ground, uneven surfaces, inclines/declines, and turns—where conventional methods frequently misclassify terrain. Consequently, gait transitions exhibit enhanced safety and smoothness, establishing a scalable intelligent control paradigm to improve mobility autonomy for individuals with motor impairments.

Technology Category

• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Machine Learning: Imitation Learning & Inverse Reinforcement Learning

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Rehabilitation technology is a natural setting to study the shared learning and decision-making of human and machine agents. In this work, we explore the use of Hierarchical Reinforcement Learning (HRL) to develop adaptive control strategies for lower-limb exoskeletons, aiming to enhance mobility and autonomy for individuals with motor impairments. Inspired by prominent models of biological sensorimotor processing, our investigated HRL approach breaks down the complex task of exoskeleton control adaptation into a higher-level framework for terrain strategy adaptation and a lower-level framework for providing predictive information; this latter element is implemented via the continual learning of general value functions (GVFs). GVFs generated temporal abstractions of future signal values from multiple wearable lower-limb sensors, including electromyography, pressure insoles, and goniometers. We investigated two methods for incorporating actual and predicted sensor signals into a policy network with the intent to improve the decision-making capacity of the control system of a lower-limb exoskeleton during ambulation across varied terrains. As a key result, we found that the addition of predictions made from GVFs increased overall network accuracy. Terrain-specific performance increases were seen while walking on even ground, uneven ground, up and down ramps, and turns, terrains that are often misclassified without predictive information. This suggests that predictive information can aid decision-making during uncertainty, e.g., on terrains that have a high chance of being misclassified. This work, therefore, contributes new insights into the nuances of HRL and the future development of exoskeletons to facilitate safe transitioning and traversing across different walking environments.

Problem

• Develop adaptive control for lower-limb exoskeletons using HRL
• Enhance mobility for individuals with motor impairments
• Improve decision-making with predictive sensor signals

Innovation

• Hierarchical Reinforcement Learning for exoskeleton control
• General Value Functions predict sensor signals
• Policy network integrates actual and predicted data