Generation as Auxiliary Supervision: Enhancing Visual Understanding at Zero Inference Overhead via Decoupled Embedding Prediction

2026-08-12

AI Summary

Existing multimodal large models typically treat visual understanding and generation as disjoint processes, limiting the potential of generation tasks to enhance comprehension capabilities. This work proposes the GAS framework, which innovatively reformulates visual generation as a Next Embedding Prediction (NEP) task and introduces cognitively aligned generative targets as auxiliary supervisory signals. Built upon a decoupled hybrid Transformer architecture, GAS shares底层 features while maintaining separate pathways for understanding and generation. The approach consistently improves multimodal comprehension across various model scales and training stages, with particularly notable gains in perceptual and spatial reasoning tasks, all without incurring additional computational overhead during inference.

Abstract

While Multimodal Large Language Models (MLLMs) have achieved remarkable progress, visual understanding and generation are typically treated as divergent objectives. Existing unified frameworks often rely on discrete visual tokenization or diffusion objectives whose generative targets differ from the continuous representations consumed by visual understanding models, making direct transfer to enhance existing pretrained MLLMs non-trivial. In this work, we present GAS, a generation-guided training framework that reinterprets visual generation as auxiliary supervision for representation learning. Concretely, GAS adapts Next Embedding Prediction (NEP) as a cross-modal generation paradigm within a decoupled Mixture-of-Transformers (MoT) architecture. By maintaining a shared lower trunk and parallel upper layers, GAS lets generation losses enrich the shared visual pathway with finer spatial precision and stronger visual retention while shielding the upper understanding layers from direct generation gradients. To maximize this synergy, we further construct highly correlated generation tasks that demand deep cognitive grounding rather than generic synthesis alone. Across model scales and training stages, GAS improves aggregate multimodal understanding, with its most reliable gains on perception and spatial comprehension. Crucially, because the auxiliary generation branch is discarded after training, these gains incur zero inference overhead. Extensive controlled comparisons and representation-level analyses further clarify when and why generation-guided training benefits understanding, and demonstrate the feasibility of generation-guided training as a practical route to stronger multimodal understanding.

Problem

• Multimodal Large Language Models
• Visual Understanding
• Visual Generation
• Auxiliary Supervision
• Zero Inference Overhead

Innovation

• Generation as Auxiliary Supervision
• Next Embedding Prediction
• Decoupled Mixture-of-Transformers
• Zero Inference Overhead
• Multimodal Understanding