Market-Bench: Evaluating Large Language Models on Introductory Quantitative Trading and Market Dynamics

2025-12-13

AI Summary

This work systematically evaluates large language models’ (LLMs) code generation capability for introductory quantitative trading tasks, focusing on end-to-end translation from natural-language strategy descriptions to executable backtesting code. To this end, we introduce MARKET-BENCH—the first benchmark tailored for beginners—covering three canonical strategies: time-based trading, pairs trading, and delta hedging. We propose a novel multi-round pass@k metric that decouples execution success from numerical accuracy, quantifying fidelity across profit-and-loss (P&L), maximum drawdown, and position trajectory error—thereby establishing a verifiable, reproducible evaluation paradigm for financial engineering practice. Experiments span 12 state-of-the-art models: GPT-5.1 Codex-Max achieves 100% pass@1 on the first two strategy classes; Qwen3 Max attains 100% pass@3. However, all models exhibit significant numerical inaccuracies in risk modeling and inventory dynamics reasoning.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Machine Learning: Large Multimodal Models (LMMs)
• Reasoning under Uncertainty: Relational Probabilistic Models

Application Category

• Economics, Online Markets and Human Computation: Uses of LLMs and GenAI for marketplace design, bidding, and strategic interactions
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

We introduce MARKET-BENCH, a benchmark that evaluates large language models (LLMs) on introductory quantitative trading tasks by asking them to construct executable backtesters from natural-language strategy descriptions and market assumptions. Each instance specifies one of three canonical strategies -- scheduled trading on Microsoft (NASDAQ: MSFT), pairs trading on Coca-Cola (NASDAQ: KO) and Pepsi (NASDAQ: PEP), or delta hedging on MSFT -- and models must produce code whose P&L, drawdown, and position paths match a verifiable reference implementation. We assess twelve state-of-the-art models using a multi-round pass@k metric that separates structural reliability (whether the backtest runs) from numerical accuracy (mean absolute error of the backtest metrics). While most models reliably execute the simplest strategy (average pass@3 of 0.80), errors vary by orders of magnitude across models and tasks: Gemini 3 Pro and Claude 4.5 Sonnet combine strong reliability with low error on simpler strategies, GPT-5.1 Codex-Max achieves perfect pass@1 on the first two strategies and the lowest best-run error on the easiest task, and Qwen3 Max attains perfect pass@3 yet sometimes produces inaccurate P&L paths. These results show that current LLMs can scaffold basic trading infrastructure but still struggle to reason robustly about prices, inventory, and risk; we release MARKET-BENCH and a public leaderboard at https://marketbench.ai.

Problem

• Evaluates LLMs on constructing executable backtesters from strategy descriptions
• Assesses model reliability and numerical accuracy in quantitative trading tasks
• Measures performance on canonical strategies like scheduled, pairs, and delta hedging

Innovation

• Benchmark for LLMs on quantitative trading tasks
• Multi-round pass@k metric for structural and numerical evaluation
• Models produce executable backtesters from natural-language descriptions