Agent Skills Can Be Harmful: An Empirical Study of Skill-Induced Failures in LLM Agents

2026-08-12

AI Summary

This work addresses the poorly understood phenomenon wherein integrating skills into large language model agents often leads to functional failures or efficiency degradation. To investigate this issue, the authors propose SkillTriage, the first attribution framework for skill-induced failures, which combines a differential analysis methodology, a structured attribution taxonomy, and automated evidence extraction. By systematically comparing agent executions with and without skills—or with semantically matched but non-functional skill variants—on the SkillsBench and SWE-Skills-Bench benchmarks, the study identifies 307 cases of skill-induced failures (125 functional failures and 182 instances of efficiency degradation). The analysis reveals that excessive validation checks and heavyweight implementation pipelines are primary root causes, thereby advocating for a new direction toward safer and more effective skill reuse in agent systems.

Abstract

Agent skills are the de facto mechanism for extending LLM agents with reusable guidance. A skill can shape the agent's task execution, including planning, tool use, problem-solving, and validation. Prior work reported mixed results of agent skills: some skills improve task success rates, while others have no effect, increase token use and execution time, and even reduce success rates. This paper presents a comprehensive analysis of skill-induced agent failures by attributing task failures and cost regressions to specific loaded skills. We introduce a differential analysis framework that attributes a failure or regression to a skill by comparing a target skill-guided run against a no-skill or semantically matched skill reference run that solves the same task, or solves it more cheaply. We instantiate this framework on SkillsBench and SWE-Skills-Bench, yielding 307 skill-induced failures, including 125 functional failures and 182 efficiency regressions. We also build SkillTriage, a taxonomy-guided attribution tool that normalizes paired cases, extracts differential evidence, and produces triage reports. Our major findings include: (1) Skill induced functional failures are rarely caused by obviously irrelevant skills; instead, seemingly relevant skills often make the agent incorrectly implement or omit task-required implementation elements. (2) Skill-induced efficiency regressions are not explained by prompt length alone. (3) The largest sources within Excessive Procedure are excessive verification and heavy implementation pipelines, contributing 67 and 30 cases, respectively. This shows that skills often turn validation checklists and construction recipes into mandatory work. Based on our findings, we propose research topics and tooling improvements for safer and more cost-aware skill reuse.

Problem

• LLM agents
• agent skills
• skill-induced failures
• efficiency regression
• functional failure

Innovation

• skill-induced failure
• differential analysis framework
• SkillTriage
• efficiency regression
• LLM agents