MT-OSC: Path for LLMs that Get Lost in Multi-Turn Conversation

2026-04-09

AI Summary

This work addresses the performance degradation, increased latency, and computational overhead that large language models (LLMs) experience in multi-turn dialogues due to excessively long contexts. To mitigate these issues, the authors propose MT-OSC, a novel lightweight dialogue history compression framework that operates without interrupting user interaction. MT-OSC employs a Condenser Agent—comprising a few-shot reasoning–driven Condenser and a lightweight Decider—to automatically and selectively preserve critical information from the conversation history in a single pass. Evaluated across 13 mainstream LLMs and multiple multi-turn benchmarks, the method reduces token usage by up to 72% over 10-turn dialogues while significantly shortening input length. Crucially, it maintains or even enhances model accuracy, effectively narrowing the performance gap observed in extended conversations.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Conversational AI/Dialog Systems
• Multiagent Systems: Agent Communication

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Large language models (LLMs) suffer significant performance degradation when user instructions and context are distributed over multiple conversational turns, yet multi-turn (MT) interactions dominate chat interfaces. The routine approach of appending full chat history to prompts rapidly exhausts context windows, leading to increased latency, higher computational costs, and diminishing returns as conversations extend. We introduce MT-OSC, a One-off Sequential Condensation framework that efficiently and automatically condenses chat history in the background without disrupting the user experience. MT-OSC employs a Condenser Agent that uses a few-shot inference-based Condenser and a lightweight Decider to selectively retain essential information, reducing token counts by up to 72% in 10-turn dialogues. Evaluated across 13 state-of-the-art LLMs and diverse multi-turn benchmarks, MT-OSC consistently narrows the multi-turn performance gap - yielding improved or preserved accuracy across datasets while remaining robust to distractors and irrelevant turns. Our results establish MT-OSC as a scalable solution for multi-turn chats, enabling richer context within constrained input spaces, reducing latency and operational cost, while balancing performance.

Problem

• multi-turn conversation
• large language models
• context window
• chat history
• performance degradation

Innovation

• multi-turn conversation
• context condensation
• large language models
• token reduction
• Condenser Agent