Advancing Visual Large Language Model for Multi-granular Versatile Perception

2025-07-22

AI Summary

Existing vision-language large models (VLLMs) are largely constrained to narrow combinations of prediction types (e.g., classification, bounding box) and instruction formats, limiting generalization and task coverage. To address this, we propose MVP-LM—a unified framework supporting both token-level and sentence-level perception, as well as bounding-box and mask prediction, enabling multi-granularity, multi-task visual understanding. Our key contributions are: (1) a chain-of-thought-inspired multi-granularity decoder that explicitly models hierarchical reasoning from fine- to coarse-grained outputs; and (2) a unified supervised fine-tuning paradigm grounded in VLLM-generated supervision, integrating query enhancement and cross-task instruction alignment. Evaluated on benchmarks including panoptic segmentation, object detection, and referring expression segmentation, MVP-LM achieves substantial performance gains, demonstrating superior generalization and seamless scalability across diverse vision-language tasks.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Large Vision Models
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

Perception is a fundamental task in the field of computer vision, encompassing a diverse set of subtasks that can be systematically categorized into four distinct groups based on two dimensions: prediction type and instruction type. Notably, existing researches often focus solely on a limited subset of these potential combinations, which constrains their applicability and versatility across various contexts. In response to this challenge, we present MVP-LM, a Multi-granular and Versatile Perception framework incorporating Visual Large Language Model. Our framework is designed to integrate both word-based and sentence-based perception tasks alongside box and mask predictions within a single architecture. MVP-LM features an innovative multi-granularity decoder in conjunction with a CoT-inspired dataset unification strategy, enabling seamless supervised fine-tuning across a wide spectrum of tasks, including but not limited to panoptic segmentation, detection, grounding, and referring expression segmentation. Furthermore, we introduce a query enhancement strategy aimed at harnessing the decoding and generative capabilities inherent in VLLMs. Extensive experiments conducted across a range of benchmarks in both word-based and sentence-based perception tasks substantiate the efficacy of our framework. The code will be available at https://github.com/xiangwentao666/MVP-LM.

Problem

• Addresses limited versatility in visual perception tasks
• Integrates word-based and sentence-based perception tasks
• Enhances decoding and generative capabilities in VLLMs

Innovation

• Multi-granular decoder for versatile perception tasks
• CoT-inspired dataset unification for supervised fine-tuning
• Query enhancement strategy leveraging VLLM capabilities