Reinforcement Learning for Risk-Sensitive Investment Management: a Free Energy--Entropy Duality Approach

2026-06-18

AI Summary

This study addresses a risk-sensitive benchmarked asset allocation problem in continuous time, where the state is uncontrollable and the terminal payoff involves a controlled Itô integral. Leveraging the free energy–entropy duality, the problem is reformulated as a linear-quadratic Gaussian stochastic differential game under an equivalent measure—a dual framework introduced here for the first time in risk-sensitive investment management. The authors derive saddle-point solutions for both finite and infinite horizons. Furthermore, they design a continuous-time actor-critic Q-learning algorithm, endowing the learned policy with an economic interpretation via fractional Kelly decomposition. Empirical results on calibrated U.S. equity market data demonstrate high learning accuracy and reveal a favorable asymmetric learning property: the portfolio actor receives a clearer learning signal than the adversarial auxiliary actor.

Abstract

This paper develops a reinforcement-learning approach to continuous-time risk-sensitive benchmarked asset allocation in a partly model-based setting. The benchmarked problem does not directly fit the standard Markovian stochastic-control template: the state is uncontrolled, whereas the terminal reward contains a controlled Itô integral. We use free energy-entropy duality to reformulate the problem as a linear-quadratic-Gaussian stochastic differential game under an equivalent probability measure, yielding explicit finite- and infinite-horizon saddle-point solutions. This structure guides a continuous-time $q$-learning actor-critic method: the quadratic value function motivates the critic, while the affine saddle-point controls motivate deterministic actors for the portfolio allocation and adversarial control. The learned allocation admits an economic interpretation through fractional Kelly decompositions. A proof-of-concept implementation calibrated to U.S. equity data shows that the actors learn the optimal policy with high accuracy and reveals a favorable asymmetry: the portfolio actor receives a cleaner learning signal than the auxiliary adversarial actor.

Problem

• risk-sensitive investment
• benchmarked asset allocation
• continuous-time reinforcement learning
• stochastic control
• Itô integral

Innovation

• free energy-entropy duality
• risk-sensitive reinforcement learning
• stochastic differential game
• actor-critic algorithm
• fractional Kelly decomposition