Factors affecting the in-context learning abilities of LLMs for dialogue state tracking

2025-06-10

AI Summary

This study investigates the key factors governing in-context learning (ICL) efficacy of large language models (LLMs) for dialogue state tracking (DST). Addressing two critical components—exemplar selection and prompt design—we propose a sentence-embedding-based k-nearest-neighbor exemplar retrieval method to ensure both semantic relevance and diversity among demonstrations, and introduce a DST-optimized templated prompt structure. Evaluations on MultiWoZ 2.4 with OLMo-7B-Instruct, Mistral-7B-Instruct-v0.3, and Llama-3.2-3B-Instruct demonstrate that exemplar relevance, semantic diversity, and structured prompting significantly improve zero-shot DST performance. To our knowledge, this is the first systematic analysis identifying the decisive factors governing ICL effectiveness in DST. Our findings provide both theoretical insights and practical guidelines for developing efficient, reproducible zero-shot DST systems without fine-tuning.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval
• Search and Optimization: Learning to Search

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

This study explores the application of in-context learning (ICL) to the dialogue state tracking (DST) problem and investigates the factors that influence its effectiveness. We use a sentence embedding based k-nearest neighbour method to retrieve the suitable demonstrations for ICL. The selected demonstrations, along with the test samples, are structured within a template as input to the LLM. We then conduct a systematic study to analyse the impact of factors related to demonstration selection and prompt context on DST performance. This work is conducted using the MultiWoZ2.4 dataset and focuses primarily on the OLMo-7B-instruct, Mistral-7B-Instruct-v0.3, and Llama3.2-3B-Instruct models. Our findings provide several useful insights on in-context learning abilities of LLMs for dialogue state tracking.

Problem

• Factors affecting in-context learning for dialogue state tracking
• Demonstration selection impact on DST performance
• LLM in-context learning abilities in MultiWoZ2.4 dataset

Innovation

• Sentence embedding based k-nearest neighbor retrieval
• Structured template for demonstrations and test samples
• Systematic analysis of demonstration selection factors