Can Small Language Models Handle Context-Summarized Multi-Turn Customer-Service QA? A Synthetic Data-Driven Comparative Evaluation

2026-01-31

AI Summary

This study addresses the high deployment costs and resource constraints of large language models by investigating the capacity of small language models to maintain dialogue coherence in multi-turn customer service interactions through context summarization. The authors propose an evaluation framework that integrates historical dialogue summaries with dialogue-stage awareness, employing instruction-tuned small models and assessing performance via lexical and semantic similarity metrics, human evaluations, and LLM-as-a-judge methodologies. Experimental results indicate that certain small models achieve overall performance comparable to commercial large models, yet exhibit inconsistent capabilities in contextual alignment and coherence, thereby revealing both their practical potential and critical limitations in real-world customer service applications.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Customer-service question answering (QA) systems increasingly rely on conversational language understanding. While Large Language Models (LLMs) achieve strong performance, their high computational cost and deployment constraints limit practical use in resource-constrained environments. Small Language Models (SLMs) provide a more efficient alternative, yet their effectiveness for multi-turn customer-service QA remains underexplored, particularly in scenarios requiring dialogue continuity and contextual understanding. This study investigates instruction-tuned SLMs for context-summarized multi-turn customer-service QA, using a history summarization strategy to preserve essential conversational state. We also introduce a conversation stage-based qualitative analysis to evaluate model behavior across different phases of customer-service interactions. Nine instruction-tuned low-parameterized SLMs are evaluated against three commercial LLMs using lexical and semantic similarity metrics alongside qualitative assessments, including human evaluation and LLM-as-a-judge methods. Results show notable variation across SLMs, with some models demonstrating near-LLM performance, while others struggle to maintain dialogue continuity and contextual alignment. These findings highlight both the potential and current limitations of low-parameterized language models for real-world customer-service QA systems.

Problem

• Small Language Models
• Multi-Turn QA
• Customer-Service
• Contextual Understanding
• Dialogue Continuity

Innovation

• Small Language Models
• Context Summarization
• Multi-turn QA
• Instruction Tuning
• Conversational Evaluation