Can Reinforcement Learning Efficiently Discover Price Manipulation?

2026-07-07

AI Summary

This study investigates the capability of model-free reinforcement learning (RL) to discover profitable price manipulation strategies in financial markets under data scarcity, benchmarking it against traditional model-based approaches. Within the Almgren-Chriss framework, the authors conduct the first analysis in a discrete-time market featuring both nonlinear permanent and linear temporary price impacts, employing the Deep Deterministic Policy Gradient (DDPG) algorithm alongside an SLSQP optimizer for strategy learning and evaluation. The results demonstrate that RL effectively uncovers profitable manipulation strategies in moderate-volatility regimes, significantly outperforming model-based methods and exhibiting greater robustness to parameter estimation noise. Both approaches fail under high volatility, while model-based methods hold a slight edge in low-volatility settings. This work thus highlights both the promise and limitations of RL for tackling complex financial control problems under unknown market dynamics.

Abstract

In this paper, we investigate whether a model-free RL agent can identify and exploit price manipulation opportunities more effectively than a traditional model-based approach that assumes correct specification of the data-generating process but relies on noisy parameter estimates. We consider a single-asset market in which prices evolve according to an Almgren-Chriss framework with non-linear permanent impact and linear temporary impact. We first establish the existence of price-manipulative strategies in discrete time and compute the optimal benchmark strategy using Sequential Least Squares Quadratic Programming under full information. We then compare two finite-sample learning approaches: a model-based procedure that estimates impact parameters from simulated execution data and an agnostic RL approach based on Deep Deterministic Policy Gradient, trained directly on the same amount of data. For intermediate volatility, the RL agent successfully discovers profitable manipulative strategies without explicit knowledge of the underlying model, even when training data are quite limited. More importantly, RL consistently outperforms the model-based approach when parameter estimates are affected by sampling error, despite the latter benefiting from the correct model specification. For large volatility, all methods are unable to identify manipulation opportunities, while for small volatility, the model based approach outperforms RL. These findings highlight both the effectiveness of RL in complex control problems and the risks associated with deploying learning algorithms in financial markets without appropriate safeguards.

Problem

• Reinforcement Learning
• Price Manipulation
• Model-based Approach
• Market Impact
• Financial Markets

Innovation

• Reinforcement Learning
• Price Manipulation
• Model-free RL
• Almgren-Chriss Model
• Deep Deterministic Policy Gradient