OpenPM: Auditable Point-in-Time Evaluation for LLM Portfolio-Management Agents

2026-08-06

AI Summary

This work addresses the pervasive overestimation of performance in existing LLM-driven trading agents, which often stems from look-ahead bias, optimistic execution assumptions, and the lack of enforced risk constraints. To remedy this, the authors propose OpenPM—a transparent, real-time evaluation framework that ensures agents operate solely on information available at decision time and enforces natural language risk directives by translating them into typed, hard constraints. The framework incorporates an auditing mechanism composed of contamination certificates, cost-sensitive curves, and constraint compliance reports, alongside a hierarchical allocator reference agent designed to disentangle analyst and constructor behaviors. Evaluated on a five-tiered real-time S&P 500 data stream within fixed backtesting windows, experiments reveal that constructors yield only marginal, model-dependent improvements over an equal-weight benchmark, analyst quality outweighs constructor choice, and turnover constitutes the dominant source of trading costs.

Abstract

Large language models are increasingly used to read markets, assess risk, and allocate capital. However, reported results for LLM trading agents can be inflated by look-ahead leakage, optimistic execution, and risk mandates that are described but not enforced. We present OpenPM, an auditable point-in-time evaluation framework for LLM portfolio-management agents. In OpenPM, an agent manages a \$1M long-only book over the S\&P 500 universe using market data at five-minute intervals. Every record visible to the agent must be available at the decision time. Natural-language risk mandates are converted into typed constraints and enforced on the executed portfolio. Each run produces audit artifacts, including a contamination certificate, a cost-sensitivity curve, and a constraint-adherence report. We also build a reference agent named the tiered allocator, where typed analysts score candidates, a constructor LLM proposes weights, and a deterministic critic guarantees feasibility. We isolate constructor behavior by capturing analyst evidence once and replaying it across constructor models. In our short-window case study, stronger constructors show modest and model-dependent gains over equal weighting on the same pool, but analyst quality matters more than constructor choice, and turnover is the main cost driver. All returns are upper bounds on a single frozen window without market impact, not validated alpha.

Problem

• look-ahead leakage
• portfolio management
• auditable evaluation
• risk constraints
• LLM trading agents

Innovation

• auditable evaluation
• point-in-time simulation
• LLM portfolio management
• risk constraint enforcement
• look-ahead bias mitigation