Efficient Hypergradient Descent for Inverse Reinforcement Learning

2026-08-11

AI Summary

This work addresses the high computational cost in bilevel optimization for inverse reinforcement learning, which arises from the need to compute the inverse of the Hessian of the inner objective. Leveraging the key observation that, at policy optimality, this Hessian is proportional to the Fisher information matrix, the authors propose an efficient natural hypergradient approximation method. By employing streaming spectral sketching, the approach implicitly approximates inverse Fisher-vector products without explicitly constructing or storing the large-scale Fisher matrix. The method achieves high-quality reward recovery and strong policy performance across both discrete and continuous control tasks, while substantially reducing memory consumption and improving computational efficiency.

Abstract

Inverse reinforcement learning (IRL) aims to recover a reward function under which the resulting policy reproduces the behavior observed in expert demonstrations. A natural approach is to formulate IRL as a bilevel optimization problem, in which the inner level corresponds to policy optimization under the learned reward and the outer level measures the discrepancy between the induced policy and expert data. However, this formulation is computationally challenging in practice because the outer update requires a hypergradient involving an inverse-Hessian-vector product for the inner objective. We address this challenge by showing that, at the inner optimum, the Hessian of the inner objective is proportional to the Fisher information matrix of the policy, yielding a structured Fisher-based hypergradient closely related to Natural Hypergradient Descent. To address the resulting scalability bottleneck associated with large Fisher matrices, we approximate the required inverse-Fisher-vector product using a streaming spectral sketch, avoiding explicit construction of the Fisher matrix. We evaluate our approach against a first-order stochastic bilevel baseline across discrete- and continuous-control environments. The results demonstrate competitive policy performance and strong reward-ranking quality, while Fisher sketching reduces curvature-storage complexity and can improve computational efficiency relative to an explicit Fisher solver.

Problem

• Inverse Reinforcement Learning
• Bilevel Optimization
• Hypergradient
• Fisher Information Matrix
• Computational Complexity

Innovation

• Hypergradient Descent
• Inverse Reinforcement Learning
• Fisher Information Matrix
• Bilevel Optimization
• Spectral Sketching