LiRA: Light-Robust Adversary for Model-based Reinforcement Learning in Real World

2024-09-29

arXiv.org

AI Summary

Model-based reinforcement learning (MBRL) suffers from a trade-off between robustness and control performance on real robots under unobservable disturbances. Method: We propose LiRA, a lightweight robust adversarial framework that introduces the novel concept of “light robustness” — a variational derivation-based constraint that adaptively modulates adversarial perturbation strength to maximize robustness within bounded, controllable performance degradation. LiRA integrates MBRL, variational adversarial training, and real-time force-feedback control, enabling end-to-end training with less than two hours of real-world, few-sample data. Contribution/Results: Simulation studies validate LiRA’s balanced robustness-performance trade-off; experiments on a quadruped robot demonstrate high-reliability force-responsive gait control and significantly improved deployment stability under physical disturbances.

Technology Category

• Machine Learning: Adversarial Learning & Robustness
• Intelligent Robots: Learning & Optimization for ROB
• Computer Vision: Adversarial Attacks & Robustness

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Model-based reinforcement learning has attracted much attention due to its high sample efficiency and is expected to be applied to real-world robotic applications. In the real world, as unobservable disturbances can lead to unexpected situations, robot policies should be taken to improve not only control performance but also robustness. Adversarial learning is an effective way to improve robustness, but excessive adversary would increase the risk of malfunction, and make the control performance too conservative. Therefore, this study addresses a new adversarial learning framework to make reinforcement learning robust moderately and not conservative too much. To this end, the adversarial learning is first rederived with variational inference. In addition, extit{light robustness}, which allows for maximizing robustness within an acceptable performance degradation, is utilized as a constraint. As a result, the proposed framework, so-called LiRA, can automatically adjust adversary level, balancing robustness and conservativeness. The expected behaviors of LiRA are confirmed in numerical simulations. In addition, LiRA succeeds in learning a force-reactive gait control of a quadrupedal robot only with real-world data collected less than two hours.

Problem

• Balancing robustness and conservativeness in adversarial learning
• Improving model-based RL for real-world robotic applications
• Automatically adjusting adversary level to avoid excessive conservatism

Innovation

• Adversarial learning with variational inference
• Light robustness constraint for performance balance
• Automatic adversary level adjustment in LiRA