OpenSkillEval: Automatically Auditing the Open Skill Ecosystem for LLM Agents

2026-05-22

AI Summary

This work addresses the lack of effective evaluation mechanisms for skill quality, model–framework compatibility, and cost-effectiveness in current open-source skill ecosystems for large language model (LLM) agents. To bridge this gap, we propose OpenSkillEval, the first dynamic, task-driven evaluation framework tailored to LLM agent skill ecosystems. Our approach automatically generates over 600 task instances from real-world artifacts and integrates more than 30 community-contributed skills across five application domains, including web design and data visualization, under a unified evaluation protocol. Experimental results demonstrate that skill availability does not necessarily imply effectiveness; performance gains are highly dependent on specific model–framework combinations, and most popular skills fail to significantly outperform a no-skill baseline. These findings provide empirical guidance for informed skill selection and deployment.

Abstract

Skills, i.e., structured workflow instructions distilled for large language models (LLMs), are becoming an increasingly important mechanism for improving agent performance on real-world downstream tasks. However, as the open-source skill ecosystem rapidly expands, it remains unclear how different models and agent frameworks interact with skills, how to evaluate skill quality, and how users should select skills under practical cost-performance trade-offs. In this paper, we present \textsc{OpenSkillEval}, an automatic evaluation framework for both skill-augmented agent systems and the skills themselves. Instead of relying on static benchmarks, \textsc{OpenSkillEval} automatically constructs realistic task instances from evolving real-world artifacts across five categories of downstream applications: presentation generation, front-end web design, poster generation, data visualization, and report generation. It further collects and organizes community-contributed skills for controlled comparison under unified task settings. Using more than 600 dynamically generated task instances and 30 open-source skills, we conduct a systematic evaluation of state-of-the-art models and agent frameworks. Our results show that skill availability does not guarantee effective skill usage, that the benefit of skill augmentation depends strongly on both the underlying model and the agent framework, and that many publicly popular skills do not consistently outperform base agents without skills. These findings highlight the need for dynamic, task-grounded evaluation and provide practical insights into the design, selection, and deployment of skills for LLM agents. Additional cases and benchmark resources are available on the project website: https://yingjiahao14.github.io/OpenSkillEval-Web/.

Problem

• skill evaluation
• LLM agents
• open-source skills
• cost-performance trade-offs
• agent frameworks

Innovation

• OpenSkillEval
• skill-augmented agents
• dynamic evaluation
• LLM skills
• task-grounded benchmarking