SOMA: Efficient Multi-turn LLM Serving via Small Language Model

2026-05-11

AI Summary

This work addresses the high latency, memory overhead, and substantial API costs incurred by repeatedly concatenating full conversation histories in multi-turn dialogues, which hinders the balance between response quality and inference efficiency. To mitigate this, the authors propose a lightweight dynamic switching mechanism that initially estimates a local response manifold and subsequently activates a compact proxy model for ensuing turns. Stable transitions are achieved through semantic discrepancy-aware alignment, augmented with anti-degradation training, localized LoRA fine-tuning, and dynamic gating, while also enabling drift rollback when necessary. Experimental results demonstrate that the proposed approach significantly reduces latency, memory consumption, and API invocation costs without compromising response quality.

Abstract

Large Language Models (LLMs) are increasingly deployed in multi-turn dialogue settings where preserving conversational context across turns is essential. A standard serving practice concatenates the full dialogue history at every turn, which reliably maintains coherence but incurs substantial cost in latency, memory, and API expenditure, especially when queries are routed to large proprietary models. Existing approaches often struggle to balance the trade-off between response quality and efficiency. We propose a framework that exploits the early turns of a session to estimate a local response manifold and then adapt a smaller surrogate model to this local region for the remainder of the conversation. Concretely, we learn soft prompts that maximize semantic divergence between the large and surrogate small language models' responses to surface least-aligned local directions, stabilize training with anti-degeneration control, and distill the mined cases into localized LoRA fine-tuning so the surrogate runs without prompts at inference. A simple gate enables a one-time switch with rollback on drift. We further provide a theoretical analysis for key components in SOMA. Extensive experiments show the effectiveness of SOMA. The source code is provided at: https://github.com/LabRAI/SOMA.

Problem

• multi-turn dialogue
• large language models
• serving efficiency
• context preservation
• response quality

Innovation

• efficient LLM serving
• small language model adaptation
• local response manifold
• LoRA fine-tuning
• multi-turn dialogue