Pixel-SAIL: Single Transformer For Pixel-Grounded Understanding

2025-04-14

AI Summary

Existing multimodal large language models (MLLMs) rely on external components—such as CLIP encoders or dedicated segmentation modules—leading to architectural complexity and limited scalability. Method: We propose the first end-to-end, pixel-level vision-language understanding model built entirely upon a single Transformer architecture, eliminating all auxiliary modules. Key technical innovations include: (1) a learnable upsampling module for high-resolution feature reconstruction; (2) early fusion of visual prompts to enable fine-grained spatial localization; (3) a vision-expert distillation mechanism to enhance cross-modal reasoning; and (4) PerBench—the first benchmark specifically designed for pixel-level understanding. Results: Our model achieves performance on par with or superior to multi-component state-of-the-art methods across four referring segmentation tasks, one visual prompting VQA task, and PerBench—while using fewer parameters and a simpler pipeline. This work establishes, for the first time, the feasibility of unified pixel-level multimodal understanding within a single-Transformer framework.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Architectures and workflows that use LLMs for crowd work

Abstract

Multimodal Large Language Models (MLLMs) achieve remarkable performance for fine-grained pixel-level understanding tasks. However, all the works rely heavily on extra components, such as vision encoder (CLIP), segmentation experts, leading to high system complexity and limiting model scaling. In this work, our goal is to explore a highly simplified MLLM without introducing extra components. Our work is motivated by the recent works on Single trAnsformer as a unified vIsion-Language Model (SAIL) design, where these works jointly learn vision tokens and text tokens in transformers. We present Pixel-SAIL, a single transformer for pixel-wise MLLM tasks. In particular, we present three technical improvements on the plain baseline. First, we design a learnable upsampling module to refine visual token features. Secondly, we propose a novel visual prompt injection strategy to enable the single transformer to understand visual prompt inputs and benefit from the early fusion of visual prompt embeddings and vision tokens. Thirdly, we introduce a vision expert distillation strategy to efficiently enhance the single transformer's fine-grained feature extraction capability. In addition, we have collected a comprehensive pixel understanding benchmark (PerBench), using a manual check. It includes three tasks: detailed object description, visual prompt-based question answering, and visual-text referring segmentation. Extensive experiments on four referring segmentation benchmarks, one visual prompt benchmark, and our PerBench show that our Pixel-SAIL achieves comparable or even better results with a much simpler pipeline. Code and model will be released at https://github.com/magic-research/Sa2VA.

Problem

• Simplifies MLLMs by removing extra components like vision encoders
• Enhances pixel-level understanding with a single transformer model
• Improves fine-grained feature extraction via vision expert distillation

Innovation

• Learnable upsampling module for visual tokens
• Visual prompt injection for early fusion
• Vision expert distillation for feature extraction