TAPS: Tool-Augmented Personalisation via Structured Tagging

2025-06-25

AI Summary

This study addresses the limited capability of large language models (LLMs) to perform personalized tool invocation in goal-oriented dialogue. To this end, we propose a fine-grained tool selection method that explicitly incorporates user preferences. Methodologically, we design a structured, label-based tool taxonomy and introduce an uncertainty-aware dynamic tool detection mechanism to enable explicit, real-time modeling of user preferences. Within a tool-augmented LLM framework, our approach jointly optimizes tool selection and execution. Experiments on the NLSI benchmark demonstrate that our method establishes new state-of-the-art performance among open-source models, significantly improving both accuracy and consistency in preference-aware tool invocation. Notably, it achieves the first interpretable and controllable fine-grained tool scheduling tailored to individual user preferences.

Technology Category

• Machine Learning: Learning Preferences or Rankings
• Natural Language Processing: (Large) Language Models
• Humans and AI: Learning Human Values and Preferences

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Recent advancements in tool-augmented large language models have enabled them to interact with external tools, enhancing their ability to perform complex user tasks. However, existing approaches overlook the role of personalisation in guiding tool use. This work investigates how user preferences can be effectively integrated into goal-oriented dialogue agents. Through extensive analysis, we identify key weaknesses in the ability of LLMs to personalise tool use. To this end, we introduce ame, a novel solution that enhances personalised tool use by leveraging a structured tagging tool and an uncertainty-based tool detector. TAPS significantly improves the ability of LLMs to incorporate user preferences, achieving the new state-of-the-art for open source models on the NLSI task.

Problem

• Integrating user preferences into goal-oriented dialogue agents
• Improving LLMs' ability to personalize tool use
• Enhancing tool-augmented models with structured tagging

Innovation

• Structured tagging tool for personalisation
• Uncertainty-based tool detector
• Enhances LLM user preference integration