OMuleT: Orchestrating Multiple Tools for Practicable Conversational Recommendation

2024-11-28

arXiv.org

AI Summary

This work addresses practical challenges in real-world conversational recommendation systems (CRS) for free-text dialogue. We propose an LLM-driven, multi-tool orchestration framework that dynamically coordinates over ten production-grade tools—including knowledge base queries and multimodal API gateways—to deliver real-time recommendations balancing relevance, novelty, and diversity. To our knowledge, this is the first CRS approach to scale heterogeneous tool integration systematically across four critical dimensions: user intent understanding, cross-tool coordination, end-to-end evaluation, and internal deployment validation. Evaluated on a real-user dataset, our framework improves recommendation relevance by 32%, novelty by 2.1×, and diversity by 47% over baseline LLMs. An alpha version has been successfully deployed in production, establishing a reusable engineering paradigm for tool-augmented CRS.

Technology Category

• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Conversational AI/Dialog Systems

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

In this paper, we present a systematic effort to design, evaluate, and implement a realistic conversational recommender system (CRS). The objective of our system is to allow users to input free-form text to request recommendations, and then receive a list of relevant and diverse items. While previous work on synthetic queries augments large language models (LLMs) with 1-3 tools, we argue that a more extensive toolbox is necessary to effectively handle real user requests. As such, we propose a novel approach that equips LLMs with over 10 tools, providing them access to the internal knowledge base and API calls used in production. We evaluate our model on a dataset of real users and show that it generates relevant, novel, and diverse recommendations compared to vanilla LLMs. Furthermore, we conduct ablation studies to demonstrate the effectiveness of using the full range of tools in our toolbox. We share our designs and lessons learned from deploying the system for internal alpha release. Our contribution is the addressing of all four key aspects of a practicable CRS: (1) real user requests, (2) augmenting LLMs with a wide variety of tools, (3) extensive evaluation, and (4) deployment insights.

Problem

• Dialogue Recommendation Systems
• User Query Understanding
• Diverse and Relevant Recommendations

Innovation

• OMuleT
• Multi-tool Collaboration
• Enhanced Recommendation