OmniStream: Mastering Perception, Reconstruction and Action in Continuous Streams

2026-03-12

AI Summary

Existing vision foundation models exhibit fragmented capabilities in semantic, temporal, and geometric perception, hindering real-time, streaming general-purpose visual understanding. This work proposes a unified streaming vision backbone that enables per-frame online processing through causal spatiotemporal attention, 3D rotational position encoding (3D-RoPE), and persistent key-value caching. Furthermore, it introduces a multi-task collaborative pretraining framework that jointly learns static semantics, dynamic changes, 3D geometry, and language alignment. For the first time, this approach achieves unified generalization of semantic, spatial, and temporal reasoning within a single frozen backbone, matching the performance of specialized models across diverse tasks—including image and video understanding, streaming 3D reconstruction, complex spatiotemporal reasoning, and robotic manipulation—thus representing a critical step toward general-purpose visual agents.

Technology Category

• Computer Vision: Large Vision Models
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Modern visual agents require representations that are general, causal, and physically structured to operate in real-time streaming environments. However, current vision foundation models remain fragmented, specializing narrowly in image semantic perception, offline temporal modeling, or spatial geometry. This paper introduces OmniStream, a unified streaming visual backbone that effectively perceives, reconstructs, and acts from diverse visual inputs. By incorporating causal spatiotemporal attention and 3D rotary positional embeddings (3D-RoPE), our model supports efficient, frame-by-frame online processing of video streams via a persistent KV-cache. We pre-train OmniStream using a synergistic multi-task framework coupling static and temporal representation learning, streaming geometric reconstruction, and vision-language alignment on 29 datasets. Extensive evaluations show that, even with a strictly frozen backbone, OmniStream achieves consistently competitive performance with specialized experts across image and video probing, streaming geometric reconstruction, complex video and spatial reasoning, as well as robotic manipulation (unseen at training). Rather than pursuing benchmark-specific dominance, our work demonstrates the viability of training a single, versatile vision backbone that generalizes across semantic, spatial, and temporal reasoning, i.e., a more meaningful step toward general-purpose visual understanding for interactive and embodied agents.

Problem

• streaming vision
• unified visual representation
• causal modeling
• spatiotemporal reasoning
• embodied agents

Innovation

• causal spatiotemporal attention
• 3D rotary positional embeddings
• streaming visual backbone
• multi-task pre-training
• online video processing