Problem
Research questions and friction points this paper is trying to address.
Coding Agents
Repository SKILLs
Optimization
Benchmarking
Task Complexity
Skill Issue: Lessons from Optimizing Repository SKILLs for Coding Agents
🤖 AI Summary
研究通过优化SKILL文件来提高编码代理性能，使用合并的拉取请求作为更难任务，并基于代理表现评分，发现GEPA方法有效提升了文档质量。
📝 Abstract
Coding agents increasingly read repository knowledge from SKILLs --- plain \texttt{.md} files versioned alongside the code. Recent work synthesizes these files automatically, by optimizing the document against a benchmark. A bare repository comes with no benchmark, and the synthetic tasks prior work builds are small enough that a capable agent saturates them with no document at all. We mine harder tasks --- merged pull requests of the repository, reverted at a single frozen base commit; and score a candidate document by whether the same agent does better with it than without it. On three Kotlin repositories, the documents GEPA finds raise this score by $4.9$pp on average, and the ones SkillOpt finds leave it where it started, $0.1$pp above the seed. The GEPA gain matches what prior work reports with the same optimizer, and at the dataset size a single repository supplies it cannot be separated from the agent's run-to-run variance; settling that would take more tasks than one repository's history yields. The documents themselves read better than the score: a maintainer of one repository found in them knowledge one only gets by working in the project.
Innovation
Methods, ideas, or system contributions that make the work stand out.
coding agents
repository SKILLs
merged pull requests
document optimization
agent performance
🔎 Similar Papers
No similar papers found.
