Intent Recognition and Out-of-Scope Detection using LLMs in Multi-party Conversations

2025-07-29

AI Summary

Intent classification and out-of-scope (OOS) detection in task-oriented dialogue systems heavily rely on large-scale annotated data, posing a critical bottleneck in low-resource settings. Method: This paper proposes a hybrid framework synergizing BERT and large language models (LLMs). Under zero-shot and few-shot settings, BERT efficiently extracts semantic features, which are structured and injected as prompt-enhanced inputs to the LLM, enabling cross-model information fusion and reasoning optimization. Contribution/Results: A lightweight feature-sharing mechanism is introduced—improving LLM generalization without increasing its parameter count. Experiments on multi-turn, multi-party dialogue datasets demonstrate that, using only 5% labeled data, our method achieves absolute accuracy gains of +12.3% for intent classification and +15.7% for OOS detection over strong baselines. The approach effectively alleviates annotation scarcity and establishes a new paradigm for robust dialogue understanding under data-constrained conditions.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Conversational AI/Dialog Systems
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling

Abstract

Intent recognition is a fundamental component in task-oriented dialogue systems (TODS). Determining user intents and detecting whether an intent is Out-of-Scope (OOS) is crucial for TODS to provide reliable responses. However, traditional TODS require large amount of annotated data. In this work we propose a hybrid approach to combine BERT and LLMs in zero and few-shot settings to recognize intents and detect OOS utterances. Our approach leverages LLMs generalization power and BERT's computational efficiency in such scenarios. We evaluate our method on multi-party conversation corpora and observe that sharing information from BERT outputs to LLMs leads to system performance improvement.

Problem

• Recognize user intents in multi-party conversations
• Detect Out-of-Scope (OOS) utterances effectively
• Reduce reliance on large annotated datasets

Innovation

• Hybrid BERT and LLMs for intent recognition
• Zero and few-shot learning settings
• Information sharing boosts performance