Let RGB Be the Language of Vision

2026-07-14

AI Summary

This work addresses the challenge that existing vision models typically rely on task-specific architectures for diverse structured prediction tasks—such as segmentation, depth estimation, and pose generation—hindering a unified approach. To overcome this limitation, the authors propose RINO, a novel framework that, for the first time, uniformly encodes various structured visual inputs and outputs (e.g., masks, depth maps, keypoints) into RGB images, thereby reformulating a wide range of vision tasks as a general RGB-to-RGB image editing problem. Built upon a shared encoder-decoder backbone, RINO establishes a language-model-like universal visual interface capable of zero-shot cross-task transfer without requiring task-specific fine-tuning. Experiments demonstrate that RINO achieves competitive zero-shot performance on both dense understanding and conditional generation tasks.

Abstract

This work introduces a unified formulation for vision models, where diverse forms of visual information beyond natural images, such as masks, depth maps, and other structured visual signals, are all represented as RGB images, while general visual tasks can be converted into a common RGB-to-RGB image editing problem. In this paradigm, different types of visual information internally share the same encoding and decoding architecture and parameters as natural images, enabling a single model to transfer across tasks through a unified visual interface, in a way analogous to how language models operate over text. We refer to this formulation as RGB In and RGB Out (RINO). Built upon a generic image editing backbone without task-specific fine-tuning, RINO demonstrates robust and competitive zero-shot performance on both dense understanding tasks such as segmentation and depth estimation (where we unify outputs as RGB), and dense-conditioned generation tasks such as pose-to-image generation (where we unify inputs as RGB). We hope this study provides useful insights toward general unified vision-language systems, where diverse visual tasks can be expressed, interpreted, and solved through a shared visual language. Code is available at https://github.com/yangtiming/RINO.

Problem

• unified vision representation
• RGB-based visual encoding
• cross-task transfer
• dense visual understanding
• visual language systems

Innovation

• RGB-to-RGB
• unified vision model
• zero-shot learning
• image editing backbone
• visual language