An Impulse Control Approach to Market Making in a Hawkes LOB Market

2025-10-30

AI Summary

This paper addresses the discrete-time optimal market-making problem in a high-fidelity limit-order book (LOB) driven by a Hawkes process, explicitly modeling queue dynamics, event clustering, and endogenous price impact. To reflect the realistic constraint that market makers cannot adjust positions continuously, we formulate a non-local Hamilton–Jacobi–Bellman quasi-variational inequality (HJB-QVI) under impulse control. To overcome the curse of dimensionality and intractability of analytical solutions, we propose a novel deep reinforcement learning framework integrating Proximal Policy Optimization (PPO) with dual neural networks, self-imitation learning, and a deep neural network solver inspired by Sirignano & Spiliopoulos (2018). Empirical results demonstrate that the trained agent achieves a Sharpe ratio exceeding 30 after minimal training—substantially outperforming benchmark strategies—and validates the framework’s efficacy in solving high-dimensional, non-local, jump-driven optimal control problems.

Technology Category

• Search and Optimization: Learning to Search
• Reasoning under Uncertainty: Sequential Decision Making
• Multiagent Systems: Adversarial Agents

Application Category

• Economics, Online Markets and Human Computation: Uses of LLMs and GenAI for marketplace design, bidding, and strategic interactions
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web

Abstract

We study the optimal Market Making problem in a Limit Order Book (LOB) market simulated using a high-fidelity, mutually exciting Hawkes process. Departing from traditional Brownian-driven mid-price models, our setup captures key microstructural properties such as queue dynamics, inter-arrival clustering, and endogenous price impact. Recognizing the realistic constraint that market makers cannot update strategies at every LOB event, we formulate the control problem within an impulse control framework, where interventions occur discretely via limit, cancel, or market orders. This leads to a high-dimensional, non-local Hamilton-Jacobi-Bellman Quasi-Variational Inequality (HJB-QVI), whose solution is analytically intractable and computationally expensive due to the curse of dimensionality. To address this, we propose a novel Reinforcement Learning (RL) approximation inspired by auxiliary control formulations. Using a two-network PPO-based architecture with self-imitation learning, we demonstrate strong empirical performance with limited training, achieving Sharpe ratios above 30 in a realistic simulated LOB. In addition to that, we solve the HJB-QVI using a deep learning method inspired by Sirignano and Spiliopoulos 2018 and compare the performance with the RL agent. Our findings highlight the promise of combining impulse control theory with modern deep RL to tackle optimal execution problems in jump-driven microstructural markets.

Problem

• Optimizing market making strategies in high-frequency limit order book markets
• Solving high-dimensional control problems with discrete trading interventions
• Overcoming computational challenges in jump-driven microstructure models

Innovation

• Impulse control framework for discrete market interventions
• Reinforcement Learning approximation using PPO architecture
• Deep learning method for solving Hamilton-Jacobi-Bellman QVI