CLQT: A Closed-Loop, Cost-Aware, Strategy-Consistent Benchmark for Diagnostic Evaluation of LLM Portfolio-Management Agents

2026-06-29

AI Summary

This work addresses critical limitations in evaluating large language model (LLM)-based portfolio management agents, which often rely on fixed-window return rankings susceptible to market path dependence and look-ahead bias while neglecting strategic consistency and reasoning plausibility. To overcome these issues, the authors propose CLQT—a diagnostic evaluation framework that enables traceable, fine-grained assessment of LLM agents through a closed-loop, cost-aware five-phase trading cycle: Collect, Synthesize, Allocate, Execute, and Reflect. The framework introduces several innovations, including temporal gating, strategy consistency scoring, a tripartite memory architecture, a role-constrained committee mechanism, and coherence metrics derived from external LLM evaluators. Integrated with hash-chain audit trails, institutional-grade cost modeling, and a five-dimensional capability scorecard (APM-CS), CLQT effectively disentangles genuine agent competence from stochastic noise in contaminated backtests, ablation studies, and live trading experiments, yielding scalable agent capability profiles rather than simplistic rankings.

Abstract

LLM agents are increasingly cast as autonomous portfolio managers, and benchmarks have moved from financial question-answering to sequential trading. Yet most still rank agents by returns over a fixed window -- a weak proxy, since a period's return is dominated by the market path and apparent alpha can dissolve once look-ahead leakage is controlled. Such a ranking certifies neither sound reasoning, nor a consistent strategy, nor a durable edge. We introduce CLQT, which reframes closed-loop trading evaluation as diagnosis rather than ranking: an instrument that localizes where and why an agent's process succeeds or fails. CLQT is a fully closed-loop, cost-aware, strategy-consistent, temporally-gated environment whose agents run a five-stage cycle: gather, synthesize, allocate, execute, reflect. Each round emits a complete DecisionRound sealed into a recompute-verifiable hash chain, so every metric is reconstructable from the trail. Six pillars form the substrate: a hard TimeGate, institutional transaction- and financing-cost modeling, strategy-consistency scoring, three-tier memory, a Model-Context-Protocol tool layer, and mandate-aware synthesis. The same agent runs as a constrained committee of specialized roles or a single full-autonomy orchestrator, making process scaffolding an experimental variable. From the audit trail we compute a five-axis capability scorecard (APM-CS: Coherence, Acuity, Composure, Discipline, Reliability), with Coherence judged partly by a held-out, out-of-cohort LLM to curb self-preference bias. We validate it on a contamination-controlled multi-model backtest with an ablation grid and a live broker track on unseen, post-cutoff data, against a repeated-run noise floor. CLQT separates outcome from capability, yielding not a model ranking but a durable, extensible map of agent competencies and limitations.

Problem

• LLM agents
• portfolio management
• benchmarking
• diagnostic evaluation
• strategy consistency

Innovation

• closed-loop evaluation
• cost-aware trading
• strategy consistency
• decision audit trail
• capability scorecard