Problem
Research questions and friction points this paper is trying to address.
tool-integrated agents
multi-objective optimization
Pareto optimality
alignment
tool-use efficiency
Towards Pareto-Optimal Tool-Integrated Agents with Pareto Ranking Policy Optimization
🤖 AI Summary
Existing alignment methods for tool-augmented large language models overly prioritize task accuracy while neglecting auxiliary objectives such as efficiency, thereby failing to achieve balanced multi-objective optimization. This work proposes ParetoPO, a novel framework that, for the first time, integrates dynamic Pareto front tracking with a ranking-based credit assignment mechanism. In the first phase, hypervolume-guided dynamic scalarization adaptively adjusts reward weights; in the second phase, Pareto ranking computes action-level advantages to optimize non-dominated trajectories. Evaluated on mathematical reasoning and multi-hop question answering tasks, the method significantly outperforms static and heuristic baselines, consistently yielding superior accuracy–efficiency trade-off policies.
📝 Abstract
Recent advances in tool-integrated language agents have significantly improved their ability to solve complex reasoning tasks. However, existing alignment methods predominantly focus on maximizing task accuracy, while overlooking auxiliary objectives such as tool-use efficiency, which are essential for practical deployment. To address this gap, we introduce ParetoPO, a two-stage multi-objective optimization framework for aligning tool-using large language models (LLMs) under competing objectives. In the first stage, ParetoPO leverages hypervolume-guided dynamic scalarization to adapt reward weights based on global Pareto frontier progress. In the second stage, it replaces scalarized learning signals with Pareto-ranking-based advantage computation, promoting nondominated trajectories through dominance-aware credit assignment. This design enables fine-grained, action-level optimization across multiple conflicting objectives. Experimental results on mathematic reasoning and multi-hop QA tasks show that ParetoPO consistently discovers policies with superior accuracy-efficiency trade-offs compared to static and heuristic baselines.
Innovation
Methods, ideas, or system contributions that make the work stand out.
Pareto Optimization
Multi-objective Reinforcement Learning
Tool-Integrated Agents
Dynamic Scalarization
Pareto Ranking
🔎 Similar Papers
