Plan Before You Trade: Inference-Time Optimization for RL Trading Agents

2026-05-12

AI Summary

This work proposes FPILOT, a novel framework that introduces model predictive control into financial reinforcement learning to overcome the limitations of static trading policies. Existing reinforcement learning agents typically employ fixed strategies during inference and cannot dynamically adapt based on price forecasts. In contrast, FPILOT leverages multi-step unconditional price predictions to construct return targets and performs real-time optimization of any pretrained policy at inference time—without requiring retraining. The approach is particularly effective in enhancing stochastic policies and achieves significant improvements in both cumulative returns and risk-adjusted performance metrics—including Sharpe, Sortino, and Calmar ratios—on the TradeMaster DJ30 benchmark. Moreover, the gains scale consistently with the quality of the underlying price predictions.

Abstract

Reinforcement learning agents for portfolio management are typically trained and deployed as static policies, with no mechanism for using price forecasts at inference time. We propose $\text{FPILOT}$ (**Fin**ancial **P**lugin **I**nference-time **L**earning for **O**ptimal **T**rading), a plugin inference-time optimization framework inspired by Model Predictive Control (MPC). Our key structural insight is that future prices mostly do not depend on one agent's portfolio allocation, so a suitable predictive model can produce a multi-step price trajectory without iterative action-conditioned rollouts as in typical reinforcement learning. At each decision step, we use the forecaster's predicted price trajectory to construct an allocation-based imagined return objective, and optimize the policy at inference-time before executing one step of the trade. Our framework is compatible with any pre-trained agent and adapts the policy to the forecaster's predictions without any retraining. Evaluated across five policy learning algorithms on the TradeMaster DJ30 benchmark, $\text{FPILOT}$ produces consistent improvements in total return and return-based risk-adjusted metrics (Sharpe, Sortino, Calmar), with stochastic policies benefiting more than deterministic ones. Further, using synthetic forecasts at calibrated quality levels, we show that gains consistently improve with forecaster quality, suggesting that our performance will improve based on advances in financial forecasting.

Problem

• reinforcement learning
• portfolio management
• inference-time optimization
• price forecasting
• trading agents

Innovation

• inference-time optimization
• model predictive control
• price forecasting
• portfolio management
• reinforcement learning