A Single Rewrite Suffices: Empirical Lessons from Production Skill Description Optimization

2026-06-29

AI Summary

This work addresses the challenge of routing errors in enterprise AI agent systems caused by overlapping skill descriptions, a problem traditionally mitigated through costly manual tuning. The authors propose a lightweight, automated approach that requires only a single large language model (LLM) rewrite of skill descriptions, supplemented by a small set of misrouted examples, to substantially improve routing accuracy. They introduce the training-validation F1 gap as a diagnostic metric for architectural intervention and establish a systematic evaluation framework. Evaluated on a production system with nine skills, the method achieves an average F1 score of 79.2%, matching the performance of manual tuning while reducing per-skill optimization time from 120 minutes to 3.8 minutes—a 32-fold efficiency gain. These results demonstrate that complex design yields diminishing returns, whereas a simple, streamlined solution is both effective and highly efficient.

Abstract

Enterprise AI agents route user queries to specialized skills by matching queries against natural language skill descriptions. When two skills share overlapping descriptions, the routing LLM misroutes queries, a failure we term skill collision. As agents scale to dozens of skills, manually tuning descriptions to maintain routing accuracy becomes a significant engineering bottleneck. We deploy an automated description optimization pipeline on a production enterprise group chat agent (9 skills, 372 regression cases). The pipeline produces descriptions averaging 79.2% F1, matching manually tuned descriptions at 79.4% F1 (average per-skill difference -0.20%, within the 0.78% multi-seed noise floor), while reducing per-skill engineering effort from 120 minutes to 3.8 minutes (32 times speedup). We then examine which pipeline components actually drive this match. Systematic ablation on both the production system and ToolBench (16k tools) reveals that a single LLM rewrite using any available false-positive and false-negative cases captures most of the available improvement. Other design choices we tested (iteration budget, feedback signal composition, dual editing of confused pairs, and training set size) each affect final F1 by less than 0.5%. Description optimization addresses skill collisions caused by overlapping descriptions but cannot resolve cases where two skills intended scopes genuinely overlap. We identify a diagnostic (a large train-validation F1 gap) that flags the latter cases for architectural rather than text-level intervention.

Problem

• skill collision
• description optimization
• LLM routing
• enterprise AI agents
• natural language skill descriptions

Innovation

• skill collision
• description optimization
• LLM rewriting
• query routing
• enterprise AI agents