Controllable Conversational Theme Detection Track at DSTC 12

2025-08-26

AI Summary

This paper addresses the limitations of existing topic detection methods for large-scale dialogues (e.g., customer service, sales), which rely on predefined intent taxonomies and lack user controllability. We propose a controllable topic detection framework that jointly clusters dialogue texts with large language model (LLM) representations, dynamically adjusts topic granularity via user-provided preference data, and supports flexible topic formulation and personalization. Evaluation employs a multi-dimensional metric combining automated scoring and human verification. Our key contributions are: (1) the first user-adjustable granularity paradigm for dialogue topic detection; (2) the release of the DSTC 12 Topic Detection task—including a real-world dialogue dataset, open-source system, and standardized evaluation benchmark; and (3) empirical validation across multiple participating teams, demonstrating significant improvements in detection accuracy, interpretability, and practical extensibility, while substantially reducing manual analysis effort.

Technology Category

• Natural Language Processing: Conversational AI/Dialog Systems
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Web Mining and Content Analysis: Topic discovery and tracking
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Conversational analytics has been on the forefront of transformation driven by the advances in Speech and Natural Language Processing techniques. Rapid adoption of Large Language Models (LLMs) in the analytics field has taken the problems that can be automated to a new level of complexity and scale. In this paper, we introduce Theme Detection as a critical task in conversational analytics, aimed at automatically identifying and categorizing topics within conversations. This process can significantly reduce the manual effort involved in analyzing expansive dialogs, particularly in domains like customer support or sales. Unlike traditional dialog intent detection, which often relies on a fixed set of intents for downstream system logic, themes are intended as a direct, user-facing summary of the conversation's core inquiry. This distinction allows for greater flexibility in theme surface forms and user-specific customizations. We pose Controllable Conversational Theme Detection problem as a public competition track at Dialog System Technology Challenge (DSTC) 12 -- it is framed as joint clustering and theme labeling of dialog utterances, with the distinctive aspect being controllability of the resulting theme clusters' granularity achieved via the provided user preference data. We give an overview of the problem, the associated dataset and the evaluation metrics, both automatic and human. Finally, we discuss the participant teams' submissions and provide insights from those. The track materials (data and code) are openly available in the GitHub repository.

Problem

• Automatically identifying and categorizing conversation topics
• Reducing manual effort in analyzing expansive dialog data
• Enabling controllable granularity in theme clustering via user preferences

Innovation

• Controllable theme detection via user preference data
• Joint clustering and labeling of dialog utterances
• Granularity control in conversational theme identification