Agent Mentor: Framing Agent Knowledge through Semantic Trajectory Analysis

2026-04-12

AI Summary

This work addresses the instability and performance degradation in AI agents caused by ambiguity in natural language prompts. To mitigate this issue, the authors propose an automated prompt refinement mechanism grounded in semantic trajectory analysis. By monitoring agent execution logs and extracting semantic features to detect undesirable behaviors, the system dynamically injects corrective instructions, enabling incremental, data-driven optimization of system prompts. Integrated into the open-source Agent Mentor library, this approach synergistically combines large language models, log analysis, and dynamic prompt engineering. Empirical evaluations across diverse agent configurations and benchmark tasks demonstrate significant improvements in accuracy, with particularly pronounced gains in scenarios where initial prompts exhibit semantic ambiguity.

Technology Category

• Natural Language Processing: Prompt Engineering / Prompting
• Multiagent Systems: Adversarial Agents
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Agentic search
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

AI agent development relies heavily on natural language prompting to define agents' tasks, knowledge, and goals. These prompts are interpreted by Large Language Models (LLMs), which govern agent behavior. Consequently, agentic performance is susceptible to variability arising from imprecise or ambiguous prompt formulations. Identifying and correcting such issues requires examining not only the agent's code, but also the internal system prompts generated throughout its execution lifecycle, as reflected in execution logs. In this work, we introduce an analytics pipeline implemented as part of the Agent Mentor open-source library that monitors and incrementally adapts the system prompts defining another agent's behavior. The pipeline improves performance by systematically injecting corrective instructions into the agent's knowledge. We describe its underlying mechanism, with particular emphasis on identifying semantic features associated with undesired behaviors and using them to derive corrective statements. We evaluate the proposed pipeline across three exemplar agent configurations and benchmark tasks using repeated execution runs to assess effectiveness. These experiments provide an initial exploration of automating such a mentoring pipeline within future agentic governance frameworks. Overall, the approach demonstrates consistent and measurable accuracy improvements across diverse configurations, particularly in settings dominated by specification ambiguity. For reproducibility, we released our code as open source under the Agent Mentor library.

Problem

• AI agents
• prompt ambiguity
• system prompts
• agent behavior
• semantic trajectory

Innovation

• semantic trajectory analysis
• system prompt adaptation
• agent mentoring
• corrective instruction injection
• agentic governance