DenseMLLM: Standard Multimodal LLMs are Intrinsic Dense Predictors

2026-02-15

AI Summary

Standard multimodal large language models (MLLMs) struggle to perform fine-grained dense prediction tasks directly and typically rely on task-specific decoders, which compromises their generality. This work proposes a universal, architecture-agnostic approach that enables standard MLLMs to execute dense prediction tasks—such as semantic segmentation and depth estimation—in an end-to-end manner through a multi-label, multi-task visual token supervision mechanism. By eliminating the need for custom-designed decoders, the method preserves model simplicity while fully unlocking the inherent dense perception capabilities of MLLMs. Extensive experiments demonstrate that this approach achieves highly competitive performance across multiple dense prediction and vision-language benchmarks.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Multi-modal Vision
• Natural Language Processing: (Large) Language Models

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Large language models for search

Abstract

Multimodal Large Language Models (MLLMs) have demonstrated exceptional capabilities in high-level visual understanding. However, extending these models to fine-grained dense prediction tasks, such as semantic segmentation and depth estimation, typically necessitates the incorporation of complex, task-specific decoders and other customizations. This architectural fragmentation increases model complexity and deviates from the generalist design of MLLMs, ultimately limiting their practicality. In this work, we challenge this paradigm by accommodating standard MLLMs to perform dense predictions without requiring additional task-specific decoders. The proposed model is called DenseMLLM, grounded in the standard architecture with a novel vision token supervision strategy for multiple labels and tasks. Despite its minimalist design, our model achieves highly competitive performance across a wide range of dense prediction and vision-language benchmarks, demonstrating that a standard, general-purpose MLLM can effectively support dense perception without architectural specialization.

Problem

• Multimodal Large Language Models
• Dense Prediction
• Semantic Segmentation
• Depth Estimation
• Architectural Fragmentation

Innovation

• Dense Prediction
• Multimodal LLM
• Vision Token Supervision
• Generalist Architecture
• Semantic Segmentation