extsc{FLAG-Trader}: Fusion LLM-Agent with Gradient-based Reinforcement Learning for Financial Trading

2025-02-17

AI Summary

This work addresses the challenge of multi-step, goal-directed trading decision-making in interactive financial markets. We propose a novel agent architecture that synergistically integrates large language models (LLMs) with gradient-driven reinforcement learning. Methodologically, we introduce the first approach to directly employ a partially fine-tuned LLM as a differentiable policy network, optimized end-to-end via REINFORCE policy gradients guided by trading rewards—thereby jointly enhancing linguistic understanding and financial decision-making capabilities. Our framework incorporates parameter-efficient fine-tuning, multimodal financial data representation, and an LLM-Agent design. Empirical evaluation on live-market simulation trading demonstrates statistically significant improvements in Sharpe ratio and win rate. Moreover, the architecture exhibits strong cross-task generalization across diverse downstream financial tasks, including portfolio optimization, event-driven trading, and risk forecasting.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Cognitive Modeling & Cognitive Systems: Agent Architectures
• Multiagent Systems: Multiagent Learning

Application Category

• Economics, Online Markets and Human Computation: Uses of LLMs and GenAI for marketplace design, bidding, and strategic interactions
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Large language models (LLMs) fine-tuned on multimodal financial data have demonstrated impressive reasoning capabilities in various financial tasks. However, they often struggle with multi-step, goal-oriented scenarios in interactive financial markets, such as trading, where complex agentic approaches are required to improve decision-making. To address this, we propose extsc{FLAG-Trader}, a unified architecture integrating linguistic processing (via LLMs) with gradient-driven reinforcement learning (RL) policy optimization, in which a partially fine-tuned LLM acts as the policy network, leveraging pre-trained knowledge while adapting to the financial domain through parameter-efficient fine-tuning. Through policy gradient optimization driven by trading rewards, our framework not only enhances LLM performance in trading but also improves results on other financial-domain tasks. We present extensive empirical evidence to validate these enhancements.

Problem

• Enhance LLM performance in financial trading
• Integrate linguistic processing with reinforcement learning
• Improve decision-making in interactive financial markets

Innovation

• LLM-Agent integration
• Gradient-based RL optimization
• Multimodal financial fine-tuning