SPEAR: Code-Augmented Agentic Prompt Optimization

2026-05-25

AI Summary

This work proposes SPEAR, an agent-based, free-form prompt optimization framework that addresses the limited flexibility and autonomy of existing automatic prompt engineering methods. SPEAR introduces the “code-as-action” paradigm to prompt optimization for the first time, enabling agents to autonomously write and execute Python code for structured error analysis—such as generating confusion matrices and performing error clustering—and dynamically decide optimization strategies using four tools: evaluate, execute, set-prompt, and terminate. The framework incorporates a sandboxed execution environment, an automatic rollback mechanism, and optional metric floor constraints to guarantee monotonic performance improvement. Evaluated across 13 industrial-scale LLM-as-judge tasks and multiple benchmarks, SPEAR significantly outperforms prior approaches, achieving a 0.938 accuracy on BBH-7 compared to 0.628 for GEPA and 0.484 for TextGrad. Ablation studies confirm the critical role of Python-based tool use in complex tasks.

Abstract

Automatic prompt engineering (APE) rewrites prompts to improve downstream task performance, but existing APE loops treat the optimizer itself as a fixed pipeline. We port the code-as-action paradigm of CodeAct (Wang et al., 2024a) to APE and propose SPEAR (Sandboxed Prompt Engineer with Active Roll-back), a free-form agentic optimizer with four tools -- evaluate, python, set_prompt, finish -- that decides autonomously how and when to use them. The distinctive tool is the Python sandbox: the optimizer writes and executes arbitrary Python on the current evaluation DataFrame, performing structural error analysis (confusion matrices, error clustering, per group metrics) the agent itself authors. Two guardrails turn the long-horizon agent into a monotone-improving optimizer: auto-rollback on metric regression, and an optional guard metric floor. We evaluate on three industrial LLM-as-judge suites (13 judge tasks across recruiter-intake, conversational-memory, and query-refinement systems) plus seven BBH tasks and GSM8K. SPEAR wins every industrial task on the primary metric ($κ$ 0.857 vs 0.359 on tool-selection; F1-macro 0.815 vs 0.763 on filter-relevance; $κ$ 0.254 vs 0.218 on the hardest extraction dimension). On BBH-7 SPEAR averages 0.938 accuracy vs GEPA 0.628 and TextGrad 0.484. Ablations show the Python tool is the largest single lever on complex judge tasks ($Δ\approx +0.79κ$ on the 5-class tool-selection judge, $Δ\approx +0.35κ$ on the hardest extraction dimension when removed); its irreplaceable contribution is class-pair confusion aggregation that a long-context LLM cannot extract reliably from the raw eval DataFrame.

Problem

• automatic prompt engineering
• prompt optimization
• code-augmented agent
• structural error analysis
• LLM-as-judge

Innovation

• code-as-action
• agentic prompt optimization
• Python sandbox
• automatic prompt engineering
• structural error analysis