PORTS: Preference-Optimized Retrievers for Tool Selection with Large Language Models

2026-07-03

AI Summary

This work addresses the misalignment between existing tool retrievers and large language model (LLM) training, which limits tool selection effectiveness. The authors propose a win-rate preference optimization method that leverages perplexity from a frozen LLM as a heuristic preference signal to jointly enhance the correlation between tool selection probability and downstream task performance. Additionally, they introduce a contrastive semantic loss based on docstrings to fine-tune the retriever. This study is the first to integrate preference optimization with contrastive learning for tool retrieval training, achieving efficient alignment between the retriever and the LLM. Evaluated across six datasets, two encoder types, and three LLMs, the approach significantly improves tool selection accuracy while maintaining low computational overhead and strong generalization capability.

Abstract

Integrating external tools with Large Language Models (LLMs) has emerged as a promising paradigm for accomplishing complex tasks. Since LLMs still struggle to effectively manage large tool collections, researchers have begun exploring retrieval-based methods to pre-select the most relevant options, addressing input length and latency constraints. However, existing retrievers are often misaligned with tool-calling LLMs due to their separate training processes. This paper presents PORTS, a novel odds ratio preference optimization method for training retrievers aimed at tool selection. Using a perplexity-inspired preference signal from a frozen LLM, our approach fine-tunes a retriever to find helpful tools by optimizing the correlation between the selection probabilities and the downstream performances while jointly enforcing a contrastive semantic loss between documentation strings. The versatility of PORTS and its ability to significantly improve tool selection accuracy are demonstrated through extensive experiments on six datasets, two encoder models, and three LLMs with diverse prior knowledge. With low computational demands, our alignment process facilitates generalization to new queries and tools, proving valuable for practical applications with evolving toolsets.

Problem

• tool selection
• retriever alignment
• large language models
• preference optimization
• external tools

Innovation

• preference optimization
• tool selection
• retriever alignment
• odds ratio
• contrastive semantic loss