An Efficient vLLM-Based Inference Pipeline for Unified Audio Understanding and Generation

2026-07-02

AI Summary

This work addresses the lack of native support for multimodal generation—particularly multi-layer audio tokens in speech-language models—in existing high-throughput inference engines. Building upon vLLM, the authors propose a unified end-to-end inference pipeline for joint audio understanding and generation. They innovatively extend autoregressive decoding to enable delayed-mode deinterleaving and multi-stream collaborative sampling, while integrating a GPU-resident acoustic decoder for efficient waveform synthesis. By co-scheduling conditional and unconditional requests within continuous batching, the system efficiently supports Classifier-Free Guidance (CFG) with only a 20% throughput penalty relative to non-CFG execution, achieving 80% of the latter’s throughput efficiency. This approach significantly enhances both multimodal audio generation quality and overall system performance, and the complete framework is released as open source.

Abstract

While Large Multimodal Models excel in comprehension, high-throughput inference engines lack native support for multimodal generation. This is severe in Speech Language Models, where generating multi-layered audio tokens via decoupled AR+NAR or synchronous Multi-Token Prediction (MTP) with delay-pattern interleaving conflicts with standard single-stream loops. We present a vLLM-based inference pipeline for unified speech understanding and generation. We extend autoregressive decoding to natively execute delay-pattern de-interleaving and coordinated multi-stream sampling, integrating an on-GPU acoustic decoder for end-to-end waveform synthesis. Crucially, we overcome the shared intuition that Classifier-Free Guidance (CFG) halves throughput. By co-scheduling paired conditional and unconditional requests within a continuous batch, our CFG implementation sustains 80% of non-CFG throughput, absorbing dual-request and logit merging overheads. We open-source our framework.

Problem

• multimodal generation
• speech language models
• high-throughput inference
• multi-token prediction
• classifier-free guidance

Innovation

• vLLM
• Multi-Token Prediction
• Classifier-Free Guidance
• delay-pattern de-interleaving
• on-GPU acoustic decoding