DeSRPA: Decoupled Speech Role-Playing Agent via Inference-Time Intervention

2026-06-16

AI Summary

Existing voice-based role-playing systems rely on end-to-end fine-tuning, which suffers from poor generalization and degrades the reasoning capabilities of large language models (LLMs). To address this, this work proposes a training-free, decoupled architecture that intervenes during inference with a frozen LLM through a dual-level control vector mechanism. This mechanism separately steers internal role cognition and external vocal expression, ensuring consistency in character personality and emotion while avoiding performance degradation caused by cross-modal alignment. Requiring no fine-tuning, the proposed method significantly outperforms end-to-end approaches on the SpeechRole and OmniCharacter benchmarks and achieves speech naturalness comparable to GPT-4o Audio.

Abstract

While Large Language Models (LLMs) have revolutionized text-based role-playing, creating immersive Speech Role-Playing Agents (SRPAs) requires a seamless bridge between cognitive reasoning and paralinguistic nuances. Current SRPAs primarily rely on end-to-end (E2E) fine-tuning. However, this paradigm suffers from poor generalization to unseen characters due to its reliance on role-specific data, while imposing a "modality alignment tax" that degrades intrinsic LLM reasoning capabilities. We propose DeSRPA, an agentic framework for character role play via inference-time intervention on frozen backbones. DeSRPA employs a dual-level control vector mechanism, Internal Cognitive Steering and External Expressive Rendering, to synchronize "mind" and "voice". Experiments on SpeechRole and OmniCharacter benchmarks demonstrate that DeSRPA significantly outperforms E2E baselines in personality and emotional consistency. It achieves high speech naturalness, narrowing the gap with proprietary models like GPT-4o Audio, while remaining a scalable and training-free paradigm.

Problem

• Speech Role-Playing Agent
• modality alignment tax
• generalization
• paralinguistic nuances
• end-to-end fine-tuning

Innovation

• inference-time intervention
• decoupled control
• speech role-playing agent
• frozen LLM
• dual-level control vector