Not All Skills Help: Measuring and Repairing Agent Knowledge

2026-06-13

AI Summary

This work addresses the task heterogeneity of natural language skills accumulated by large language model agents without weight updates, where certain skills can be detrimental to specific tasks—a nuance overlooked by existing methods that rely on global filtering. The authors propose ASSAY, a framework that for the first time reveals pervasive causal heterogeneity within skill repositories. By employing random masking to quantify the causal contribution of each skill on a development set, ASSAY offline reconstructs the skill library and dynamically suppresses harmful skills at inference time, decoupling generation from selection. Experiments demonstrate that ASSAY boosts DeepSeek-V3’s task completion rate to 69.3% on the hardest AppWorld split (a relative +47.4%) and enables GPT-4.1 to surpass o4-mini, o1, and GPT-4.5 on tau-bench retail tasks (a relative +8.7%), all without modifying model weights.

Abstract

LLM agents can improve without weight updates by accumulating natural-language skills from experience, but current systems entrust every decision about which skills to keep and how to apply them to LLM judgment alone. We argue that this conflates two distinct roles: generating a skill from experience is a creative act that judgment handles well, while deciding whether that skill actually helps requires empirical evidence across many tasks. Measuring per-skill causal contributions via randomized masking, we find that skill libraries exhibit pervasive causal heterogeneity: individual skills routinely help on some task types while hurting on others, yet their opposing effects cancel in aggregate, making them invisible to global curation methods. We propose ASSAY, a framework that separates generation from curation: it computes a per-skill causal attribution on a small development set, restructures the library offline, and suppresses skills with negative predicted effect for each test task. Across seven base models spanning four providers and two benchmarks (AppWorld and tau-bench), ASSAY consistently improves over prior skill-curation approaches. On AppWorld's hardest split, DeepSeek-V3 achieves 69.3% task-goal completion (47.4% relative improvement), a new state of the art among all published methods including weight-tuned approaches. On tau-bench retail, GPT-4.1 improves by 8.7% relative, advancing past o4-mini, o1, and GPT-4.5 on the public leaderboard without any weight modification. Ablation traces the dominant gain to per-task masking, confirming that the bottleneck is matching skills to tasks at inference time, not removing bad skills globally. Code is available at https://github.com/aiming-lab/assay.

Problem

• LLM agents
• skill curation
• causal heterogeneity
• task-specific effectiveness
• skill libraries

Innovation

• causal attribution
• skill curation
• LLM agents
• randomized masking
• task-specific skill selection