Learning to Control LLM Agent Harnesses with Offline Reinforcement Learning

2026-07-05

AI Summary

This work addresses the limitation of conventional LLM agent execution frameworks, which are typically treated as fixed infrastructure and thus constrain performance improvements without modifying the underlying model or prompts. The authors propose modeling the execution framework as a finite-horizon Harness MDP and introduce a lightweight, learnable control layer that selects structured execution actions via offline reinforcement learning and advantage-weighted regression, all while keeping the LLM frozen. A novel Harness Maturity Score is introduced to decouple procedural behavior from final answer quality, revealing how process-level optimization can be effectively translated into outcome gains through offline data. Experiments demonstrate that the approach significantly enhances verification behaviors across six controlled tasks and two public benchmarks—including tau-bench retail, AgentBench DB-Bench, and code generation tasks with calibrated verifiers—and selectively improves final task performance.

Abstract

Large language model (LLM) agents are usually improved by changing prompts, models, or hand-written workflows, while the execution harness around the model is treated as fixed infrastructure. We argue that this harness is itself a learnable control layer. We formalize harness operation as a finite-horizon Harness MDP, where a lightweight controller selects structural execution actions while the LLM executor remains frozen. The controller is trained from offline rollouts using advantage-weighted regression with only terminal task-rubric rewards. We also separate final task quality from a post-hoc Harness Maturity Score, which measures whether the harness follows reliable execution patterns rather than only whether the final answer is correct. This separation gives a finite-buffer view of harness learning: final-quality gains require high-return support in the offline buffer, while process behavior can shift whenever it aligns with advantage-weighted actions. Across six controlled domains and two public-benchmark adapters, the learned controller consistently improves verification behavior and selectively improves final task quality, with the largest gains on adapted tau-bench retail, adapted AgentBench DB-Bench, and coding with a calibrated structural verifier. Ablations against behavior cloning and Forced CHECK show that the gains are not explained by imitation or by simply adding checks. These results identify harness control as a learnable layer for frozen LLM agents, while showing that offline support limits when better process control becomes better final answers.

Problem

• LLM agent
• execution harness
• offline reinforcement learning
• task quality
• process control

Innovation

• offline reinforcement learning
• LLM agent harness
• advantage-weighted regression
• Harness MDP
• process-quality decoupling