Aero Realtime: Fully Aligned Input-Output Streams for Low-Latency Streaming Multimodal Generation

2026-08-09

AI Summary

This work proposes the first duplex streaming multimodal model (4B parameters) that fully aligns input and output streams, overcoming the limitations of existing non-duplex, turn-based approaches which cannot accept new inputs during generation, leading to discontinuous interaction and misalignment between response timing and language production. The model unifies video, audio, and text into a shared temporal grid with 80-millisecond slots and jointly learns response timing and content generation through a single autoregressive objective. Key innovations include slot-aligned supervision, real-time question-answering data, hardware-aware distributed training, and KV cache reuse. Evaluated on four A6000 GPUs, the system achieves a median processing latency of 84 ms and P95 latency of 173 ms, with temporal deviation consistently under 200 ms—significantly outperforming conventional polling or gating mechanisms.

Abstract

Existing streaming multimodal models process observations incrementally but still follow a turn-based prefill-then-decode pattern, making them non-duplex: new observations cannot naturally enter an active generation stream. Proactive alternatives use micro-turn polling or external response gates, which fragment continuous interaction, decouple response timing from language generation, and complicate KV-cache-friendly serving. We introduce Aero Realtime, a 4B streaming multimodal model with a duplex architecture for realtime generation. Aero Realtime aligns video, audio, and textual output on a shared temporal grid, where each approximately 80-ms audio slot predicts either a lexical token or a silence token. This allows input and output to advance together, enabling one autoregressive objective to learn both when to respond and what to generate. During inference, Aero Realtime appends only the newest multimodal slot, carries forward the previous output state, and reuses the KV cache for efficient incremental execution. We further provide a complete training and serving recipe, including realtime QA construction, slot-aligned supervision, hardware-aware distributed training, and resumable inference. On four NVIDIA A6000 workstation GPUs, Aero Realtime maintains 84-ms median and 173-ms P95 processing lag over 20 minutes of a continuously streamed video, remaining within 200~ms of the source timeline. These results demonstrate the feasibility of fully aligned input-output modeling for duplex, proactive, and hardware-aligned multimodal interaction.

Problem

• streaming multimodal generation
• duplex interaction
• low-latency
• input-output alignment
• realtime response

Innovation

• duplex streaming
• temporal alignment
• multimodal generation
• KV-cache reuse
• realtime interaction