SkillSentry: Reliable Skill Execution for LLM Agents via Runtime Assurance

2026-08-10

AI Summary

This work addresses the unreliability of large language model (LLM) agents in skill execution, which often stems from procedural deviations or step-level errors. To enhance robustness, the authors propose SkillSentry—a runtime assurance framework that dynamically guides agent behavior by integrating formal specifications extracted from skill documentation with empirical patterns mined from historical execution traces. SkillSentry introduces a novel domain-specific language (DSL) tailored for skill-oriented guidance and enables real-time monitoring, adaptive guidance generation, and iterative refinement during execution. Evaluated across 15 diverse skills, two agent architectures, and four LLMs, SkillSentry improves average task success rates by 24.1%, significantly boosting execution consistency and reducing outcome variance.

Abstract

LLM agents are increasingly equipped with skills to perform complex tasks through multi-step reasoning and tool use. Although skills provide reusable procedural knowledge, agents may still execute them unreliably. Even when an agent has demonstrated the capability to complete tasks under the guidance of a skill, it may fail to do so consistently across similar tasks or repeated runs due to deviations from the skill procedure or incorrect execution of individual steps. Such instability limits the practical reliability of LLM agents. To address this problem, we propose SkillSentry, a skill-oriented runtime assurance framework built upon a new domain-specific language (DSL) for representing runtime guidance for skill execution. SkillSentry initializes the runtime guidance by combining a skill specification extracted from the corresponding skill document with execution experience mined from historical successful and failed traces. It then wraps around the agent execution loop to monitor and guide skill execution under the current guidance, while iteratively refining the guidance using newly collected traces. We evaluate SkillSentry on 15 skills across two LLM agents, each paired with two backbone models, i.e., Claude Code with Claude-Haiku-4.5 and Claude-Opus-4.6, and Codex with GPT-5.2 and GPT-5.4. Our results show that SkillSentry improves the task success rate of LLM agents by 24.1% across skills, on average, while exhibiting lower variability across repeated runs.

Problem

• LLM agents
• skill execution
• reliability
• runtime assurance
• execution instability

Innovation

• runtime assurance
• skill execution
• domain-specific language (DSL)
• LLM agents
• execution trace mining