TRACE: TRajectory Attribution for Automated Context Engineering

2026-08-10

AI Summary

This work addresses the frequent failure of AI agents in production environments due to errors or omissions in contextual sources such as system prompts, knowledge bases, or tool descriptions—a problem exacerbated by the reliance on manual log inspection for maintenance, which does not scale. To overcome this, the authors propose an automated context engineering framework that operates without explicit user feedback by mining implicit dissatisfaction signals (e.g., corrections, rephrasings, or task abandonment) from historical interaction trajectories. The framework integrates multi-component causal attribution with an exploratory validation strategy to automatically diagnose and repair contextual defects. Key contributions include the first verifiable simulation benchmark for context debugging, a taxonomy of six failure types, and a causal attribution and active verification mechanism applicable across heterogeneous context sources. Experiments demonstrate 72.7% root-cause attribution accuracy and 82% end-to-end repair effectiveness over 60 dissatisfaction trajectories, confirming the approach’s capability for efficient self-repair of context-layer faults.

Abstract

Production AI agents fail when their context sources -- system prompts, knowledge bases, tool descriptions, and procedural skills -- contain errors or gaps. Current maintenance relies on manual log review and ad-hoc debugging, creating a scalability bottleneck as interaction volume grows. We present TRACE (TRajectory Attribution for Automated Context Engineering), an automated feedback loop that mines historical agent trajectories to diagnose and remediate context failures. Our key insight is that trajectories are rich with implicit dissatisfaction signals -- user corrections, rephrasing, abandonment cues -- that reveal precisely where context sources failed, without explicit feedback collection. Unlike model fine-tuning, TRACE operates on the context layer, enabling rapid iteration without retraining. We make four contributions: (1) a trajectory mining framework that systematically extracts diagnostic information from historical agent executions; (2) multi-component causal attribution that extends textual gradients from monolithic prompt optimization to heterogeneous context sources (skills, knowledge bases, tools, prompts); (3) exploratory verification, where agents actively read context sources to distinguish content gaps requiring CREATE from stale content requiring UPDATE, achieving 96% operation accuracy; and (4) a reusable simulation methodology and verifiable benchmark addressing the absence of open datasets for context debugging, with a six-category fault taxonomy, ground truth annotations, and a cross-layer verification protocol. On 60 dissatisfaction traces spanning three complexity tiers (up to 16 execution nodes), TRACE achieves 72.7% root cause attribution and 82% end-to-end fix effectiveness, showing that over 80% of context-layer failures can be automatically diagnosed and remediated by mining historical trajectories, an overlooked resource in production systems.

Problem

• context failure
• AI agent maintenance
• scalability bottleneck
• trajectory mining
• context debugging

Innovation

• trajectory mining
• context engineering
• causal attribution
• exploratory verification
• automated debugging