SPAR: Semantic-Pixel Self-Alignment and Adaptive Routing for Unified Multimodal Models

2026-06-22

AI Summary

This work addresses the challenge faced by multimodal large language models in simultaneously achieving robust semantic understanding and precise pixel-level reconstruction in visual generation tasks. To this end, the authors propose SPAR, a unified framework that integrates a dual-stream encoder for semantic and pixel representations, an embedded self-aligned diffusion generation paradigm, and a dynamic token routing mechanism. Notably, SPAR jointly optimizes generation and comprehension capabilities without relying on external teacher signals. The approach enables, for the first time, semantic-guided adaptive feature aggregation and pixel-stream enhancement, significantly improving both generation quality and reconstruction fidelity. Within a single unified architecture, SPAR achieves state-of-the-art performance while preserving strong foundational visual understanding capabilities.

Abstract

Multimodal Large Language Models (MLLMs) have achieved remarkable success in visual understanding but remain constrained in visual generation due to the fundamental feature discrepancy between semantic perception and pixel-level reconstruction. Bridging this gap requires overcoming two core challenges: endowing semantic encoders with high-fidelity reconstruction capabilities, and effectively aligning generative models with semantic spaces without relying on external teachers. To this end, we propose a novel unified multimodal framework featuring \textbf{S}emantic-\textbf{P}ixel self-alignment and \textbf{A}daptive \textbf{R}outing (\textbf{SPAR}). First, to reconcile semantic perception with pixel-level reconstruction, we introduce an asymmetric dual-stream unified tokenizer. A lightweight semantic stream anchors discriminative features, while a Transformer-augmented pixel stream recovers fine-grained visual details into a unified compact latent space. Second, to eliminate external dependencies, we propose a self-aligned generation paradigm that natively leverages this optimized tokenizer as an internal alignment teacher for the diffusion model. Furthermore, to facilitate flexible multimodal interaction within this unified space, we introduce Dynamic Token Routing, which enables each token to adaptively aggregate multi-layer MLLM features based on its distinct semantic demands. Extensive experiments demonstrate that SPAR establishes the state-of-the-art for unified architectures, achieving exceptional generation and reconstruction quality while preserving foundational visual understanding capabilities.

Problem

• Multimodal Large Language Models
• Semantic-Pixel Alignment
• Visual Generation
• Feature Discrepancy
• Unified Multimodal Framework

Innovation

• Semantic-Pixel Alignment
• Unified Multimodal Model
• Self-Aligned Generation
• Adaptive Token Routing
• Dual-Stream Tokenizer