PodAgent: A Comprehensive Framework for Podcast Generation

2025-03-01

AI Summary

Current automatic audio generation methods struggle to support high-quality podcast production due to shallow content depth, mismatched speaker identity assignment, and insufficient naturalness and expressiveness in speech delivery. To address these challenges, we propose an end-to-end podcast generation framework featuring three key innovations: (1) a Host-Guest-Writer multi-agent collaborative content generation mechanism; (2) a speaker identity matching pool for precise role allocation; and (3) an LLM-driven conversational TTS synthesis method. Our approach integrates multi-agent systems, large language models, fine-grained speaker modeling, and prompt-optimized speech synthesis. Experiments demonstrate that the generated content exhibits significantly greater topical depth and logical coherence than direct GPT-4 outputs; speaker-role matching accuracy reaches 87.4%; and subjective evaluations confirm superior conversational fluency, emotional expressivity, and prosodic naturalness. This work establishes a scalable paradigm for long-form, multi-speaker, highly expressive audio content generation.

Technology Category

• Natural Language Processing: Generation
• Machine Learning: Large Multimodal Models (LMMs)
• Multiagent Systems: Agent Communication

Application Category

• Search and Retrieval-Augmented AI: Agentic search
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems

Abstract

Existing Existing automatic audio generation methods struggle to generate podcast-like audio programs effectively. The key challenges lie in in-depth content generation, appropriate and expressive voice production. This paper proposed PodAgent, a comprehensive framework for creating audio programs. PodAgent 1) generates informative topic-discussion content by designing a Host-Guest-Writer multi-agent collaboration system, 2) builds a voice pool for suitable voice-role matching and 3) utilizes LLM-enhanced speech synthesis method to generate expressive conversational speech. Given the absence of standardized evaluation criteria for podcast-like audio generation, we developed comprehensive assessment guidelines to effectively evaluate the model's performance. Experimental results demonstrate PodAgent's effectiveness, significantly surpassing direct GPT-4 generation in topic-discussion dialogue content, achieving an 87.4% voice-matching accuracy, and producing more expressive speech through LLM-guided synthesis. Demo page: https://podcast-agent.github.io/demo/. Source code: https://github.com/yujxx/PodAgent.

Problem

• Generates podcast-like audio with in-depth content.
• Improves voice production for expressive conversational speech.
• Develops evaluation criteria for podcast audio generation.

Innovation

• Host-Guest-Writer multi-agent collaboration system
• Voice pool for role-voice matching
• LLM-enhanced expressive speech synthesis