MOSAIC: Adaptive Inter-layer Composition for Efficient Heterogeneous Vision-Language Models

2026-07-09

AI Summary

Existing heterogeneous vision-language models rely on handcrafted static architectures, which struggle to simultaneously achieve hardware compatibility and optimal performance. This work proposes MOSAIC, a method that, for the first time, automatically searches for the optimal combination of heterogeneous layers under hardware constraints via multi-objective mixed-integer programming, establishing a unified search space encompassing linear, sparse, and low-rank operators. To mitigate performance degradation caused by compression, MOSAIC introduces a two-stage distillation strategy: first, off-policy global distillation broadens knowledge transfer, followed by dual-teacher online distillation to stabilize the output distribution. Experiments demonstrate that MOSAIC-4B matches baseline performance while reducing training cost by 2% and accelerating inference by 1.76× during prefill and 2.54× during decoding.

Abstract

Vision-Language Models (VLMs) have achieved success using homogeneous Transformers to process multimedia data. Recent studies show that heterogeneous structures interleaving efficient mechanisms, like linear attention, improve both performance and inference latency over homogeneous designs. However, these efforts rely on handcrafted static mixing patterns, which are sub-optimal and difficult to adapt to specific hardware. To bridge this gap, we propose Multi-Objective Search for Adaptive Inter-layer Composition (MOSAIC), a hardware-aware search method that automatically transforms homogeneous models into optimized heterogeneous architectures. MOSAIC integrates diverse efficiency mechanisms--including linear, sparse, and low-rank operators--into a unified search space. By formulating the selection as a multi-objective Mixed Integer Programming (MIP) problem, our method identifies optimal configurations that maximize downstream performance under strict hardware latency constraints. To mitigate performance degradation from structural transitions, we introduce a two-stage parameter recovery process: global off-policy distillation to stabilize internal representations, followed by a dual-teacher on-policy distillation leveraging a 235B oracle for knowledge expansion and the original 4B teacher for distributional stability. We validate MOSAIC through MOSAIC-4B, derived from Qwen3-VL-4B-Instruct. Results demonstrate that MOSAIC-4B matches the baseline's performance across multiple benchmarks while requiring less than 2% of the original training cost. Furthermore, it substantially improves inference efficiency, achieving 1.76x prefilling and 2.54x decoding speedups.

Problem

• Vision-Language Models
• heterogeneous architectures
• hardware-aware optimization
• static mixing patterns
• inference latency

Innovation

• heterogeneous architecture
• hardware-aware search
• multi-objective optimization
• knowledge distillation
• efficient vision-language models