Guided by the Plan: Enhancing Faithful Autoregressive Text-to-Audio Generation with Guided Decoding

2026-01-18

AI Summary

This work addresses the challenge of maintaining semantic fidelity in autoregressive text-to-audio generation when handling complex or multi-event prompts. The study reveals that the initial prefix of the generated sequence implicitly encodes global semantic planning capabilities. Building on this insight, the authors propose Plan-Critic, a lightweight auxiliary model that leverages Generalized Advantage Estimation (GAE) to evaluate and prune generation paths early during inference, thereby guiding the model toward outputs that are both high-fidelity and semantically consistent. This approach uniquely unifies causal generation with global semantic alignment while introducing computational overhead comparable to standard best-of-N sampling. Experiments demonstrate up to a 10-point improvement in CLAP score over baseline methods, establishing a new state of the art in autoregressive text-to-audio generation.

Technology Category

• Natural Language Processing: Generation
• Planning, Routing, and Scheduling: Planning with Language Models
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Search and Retrieval-Augmented AI: Agentic search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications

Abstract

Autoregressive (AR) models excel at generating temporally coherent audio by producing tokens sequentially, yet they often falter in faithfully following complex textual prompts, especially those describing complex sound events. We uncover a surprising capability in AR audio generators: their early prefix tokens implicitly encode global semantic attributes of the final output, such as event count and sound-object category, revealing a form of implicit planning. Building on this insight, we propose Plan-Critic, a lightweight auxiliary model trained with a Generalized Advantage Estimation (GAE)-inspired objective to predict final instruction-following quality from partial generations. At inference time, Plan-Critic enables guided exploration: it evaluates candidate prefixes early, prunes low-fidelity trajectories, and reallocates computation to high-potential planning seeds. Our Plan-Critic-guided sampling achieves up to a 10-point improvement in CLAP score over the AR baseline-establishing a new state of the art in AR text-to-audio generation-while maintaining computational parity with standard best-of-N decoding. This work bridges the gap between causal generation and global semantic alignment, demonstrating that even strictly autoregressive models can plan ahead.

Problem

• autoregressive generation
• text-to-audio
• instruction following
• semantic alignment
• complex sound events

Innovation

• guided decoding
• autoregressive generation
• implicit planning
• Plan-Critic
• text-to-audio