A Noise-Robust Elicit-to-Optimize Framework for Distortion Riskmetrics via Inverse Reinforcement Learning

2026-07-15

AI Summary

This study addresses the challenge of accurately identifying an agent’s risk preferences under noisy and suboptimal behavioral observations, and leverages this understanding to optimize decision-making policies with respect to distorted risk measures. To this end, the authors propose a “distill-and-optimize” framework: first, an adaptive Bayesian inverse reinforcement learning approach robustly infers the underlying distortion risk measure from noisy trajectories, with theoretical guarantees that a finite set of discriminative problem instances suffices for unique identification at an exponential convergence rate; then, for the first time, they unify the conditional cost quantile function with the integral representation of distortion functions, enabling a quantile-network-based, model-free reinforcement learning algorithm (an extension of PPO) that achieves end-to-end policy optimization for general risk-sensitive objectives. Experiments in complex financial environments demonstrate high accuracy in risk preference identification and effective policy optimization across multiple classes of distortion risk measures.

Abstract

We propose a noise-robust elicit-to-optimize framework that integrates inverse reinforcement learning (IRL) and reinforcement learning (RL) for eliciting agents' risk preferences and optimizing policies under a broad class of risk objectives characterized by distortion riskmetrics. On the elicitation side, we propose an adaptive Bayesian IRL method that infers agents' latent risk objectives from their noisy observed decisions, explicitly allowing agents to take stochastic and suboptimal actions. We establish the existence of a finite set of distinguishing questions that identifies the preferred distortion riskmetric within the candidate class and prove that the convergence rate of the algorithm is of order $O(\exp(-cm+O(\sqrt{m\log m})))$ under general settings, where $c>0$ is a constant and $m$ denotes the number of algorithm iterations. On the optimization side, we develop a model-free RL algorithm for optimizing policies under conditional distortion riskmetrics. By representing the objective as an integral of the conditional cost quantile function with respect to the distortion function, the method unifies distortion-riskmetric objectives. We optimize diverse risk objectives by extending the Proximal Policy Optimization (PPO) algorithm with policy, value, and quantile neural networks, where the quantile network estimates the full conditional cost quantile function and enables numerical evaluation of general risk objectives. A comprehensive empirical study demonstrates the framework's elicitation accuracy and effectiveness in complex financial environments.

Problem

• distortion riskmetrics
• inverse reinforcement learning
• risk preference elicitation
• noise-robustness
• policy optimization

Innovation

• distortion riskmetrics
• inverse reinforcement learning
• noise-robust elicitation
• quantile neural network
• risk-sensitive reinforcement learning