Bilevel Optimization of Agent Skills via Monte Carlo Tree Search

2026-04-17

AI Summary

This work proposes a novel framework for jointly optimizing the skills of large language model (LLM) agents, where each skill comprises instructions, tools, and supporting resources whose structure and content are tightly coupled, resulting in a complex and interdependent optimization landscape. For the first time, this joint optimization is formalized as a bilevel problem: the outer loop employs Monte Carlo Tree Search to explore skill structures, while the inner loop optimizes the content of components given a fixed structure, with both loops leveraging LLMs to guide decision-making. By integrating structured skill representations with an efficient search strategy, the approach significantly enhances agent performance on an open-source operations research question-answering benchmark, demonstrating both the effectiveness and novelty of the proposed framework.

Technology Category

• Search and Optimization: Learning to Search
• Multiagent Systems: Multiagent Learning
• Cognitive Modeling & Cognitive Systems: Agent Architectures

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Agent \texttt{skills} are structured collections of instructions, tools, and supporting resources that help large language model (LLM) agents perform particular classes of tasks. Empirical evidence shows that the design of \texttt{skills} can materially affect agent task performance, yet systematically optimizing \texttt{skills} remains challenging. Since a \texttt{skill} comprises instructions, tools, and supporting resources in a structured way, optimizing it requires jointly determining both the structure of these components and the content each component contains. This gives rise to a complex decision space with strong interdependence across structure and components. We therefore represent these two coupled decisions as \texttt{skill} structure and component content, and formulate \texttt{skill} optimization as a bilevel optimization problem. We propose a bilevel optimization framework in which an outer loop employs Monte Carlo Tree Search to determine the \texttt{skill} structure, while an inner loop refines the component content within the structure selected by the outer loop. In both loops, we employ LLMs to assist the optimization procedure. We evaluate the proposed framework on an open-source Operations Research Question Answering dataset, and the experimental results suggest that the bilevel optimization framework improves the performance of the agents with the optimized \texttt{skill}.

Problem

• bilevel optimization
• agent skills
• Monte Carlo Tree Search
• large language models
• skill optimization

Innovation

• bilevel optimization
• Monte Carlo Tree Search
• agent skills
• large language models
• structured optimization