Omni-Flow: A Unified Workflow Orchestration and Distributed KV Cache Sharing Framework for Multimodal Inference

2026-06-29

AI Summary

Existing multimodal inference systems face challenges including rigid workflow orchestration, inefficient intermediate data transfer, and difficulty in sharing KV caches and model weights across heterogeneous components. This work proposes the first system-level unified abstraction for multimodal inference, featuring a three-tier architecture that co-optimizes control flow, data flow, and compute flow. The control flow layer employs a Python DSL to support both static and dynamic workflow orchestration; the data flow layer implements a zero-copy, distributed paged KV cache spanning GPUs, CPUs, and SSDs; and the compute flow layer enables multimodal prefix matching and KV reuse, unifying the forward passes of LLMs and diffusion models through a common SGLang interface. This design decouples orchestration logic from data transmission mechanisms, achieving efficient inference and resource reuse across diverse scenarios such as LongCat-Next dialogue and HunyuanImage-3 generation.

Abstract

As large language model (LLM) inference evolves from text-only to multimodal paradigms, inference systems face three challenges: (1) flexible orchestration of multimodal workflows, where heterogeneous computing units exhibit complex dependencies and concurrent control; (2) efficient transmission of massive intermediate data across processes and nodes, with tensors flowing at high speed among heterogeneous roles; and (3) efficient sharing of KV caches and model weights across roles to eliminate redundant GPU memory. Existing solutions deploy LLMs and diffusion models independently, lacking a system-level abstraction for multimodal pipelines; this scatters orchestration logic, tightly couples transmission paths to specific models, and incurs high cost to integrate new models. To address these challenges, we present Omni-Flow, a distributed scheduling framework for multimodal inference through a three-layer abstraction. The Control Flow layer defines workflows via a Python DSL, orchestrating heterogeneous units into a unified dataflow graph that supports static DAGs and dynamic routing, with built-in service discovery and diverse load-balancing strategies. The Data Flow layer provides a distributed KV cache abstraction beyond prefill/decode separation, unifying allocation and enabling direct cross-role transmission across a three-tier paged storage hierarchy (GPU/CPU/SSD) over zero-copy, low-latency channels. The Compute Flow layer supports complex multimodal prefix matching for KV reuse across multi-turn dialogues, and takes over KV cache and sampling logic via a unified SGLang interface, letting diffusion models directly reuse the LLM forward path under unified parallel semantics. We demonstrate that Omni-Flow supports diverse heterogeneous scenarios with a consistent programming model, including omni-modal dialogue (LongCat-Next) and complex image generation pipelines (HunyuanImage-3).

Problem

• multimodal inference
• workflow orchestration
• distributed KV cache
• heterogeneous computing
• intermediate data transmission

Innovation

• multimodal inference
• workflow orchestration
• distributed KV cache
• zero-copy transmission
• unified parallel semantics