Enhancing Part-Level Point Grounding for Any Open-Source MLLMs

2026-06-28

AI Summary

This work addresses the limited precision of existing open-source multimodal large language models (MLLMs) in part-level visual grounding, which hinders fine-grained interaction. To overcome this, the authors propose a plug-and-play framework that requires no model fine-tuning: by inserting a Q-Synth module into intermediate layers of an MLLM, the method generates text-conditioned, localization-aware queries, which are then processed by a lightweight Attention-to-Point decoder to transform the model’s intrinsic attention maps into point-centered heatmaps for accurate part-level localization. The original MLLM parameters remain frozen, fully preserving its pretrained capabilities. The approach achieves substantial gains in grounding accuracy across multiple benchmarks and can be seamlessly integrated into any open-source MLLM without architectural modifications.

Abstract

Visual grounding aims to associate free-form textual queries with specific regions in an image. While recent Multimodal Large Language Models (MLLMs) have demonstrated promising capabilities in this domain, they primarily excel at object-level grounding and often struggle with part-level grounding-an essential requirement for fine-grained tasks such as robotic manipulation. In this work, we introduce a general approach that equips any open-source MLLMs with accurate 2D part-level point grounding, offering a more direct alternative to conventional grounding representations. Our method leverages the attention mechanisms inherently present in MLLMs. By synthesizing text-conditioned, grounding-aware queries within intermediate layers via the proposed Q-Synth Module, we capture target-relevant attention patterns and refine them with a lightweight Attention-to-Point Decoder, which converts these patterns into a point-centric heatmap for final prediction. Notably, all original MLLM parameters are frozen, ensuring full preservation of their pre-trained capabilities. Experiments show that our design consistently improves part-level grounding accuracy across datasets and can be seamlessly integrated into any open-source MLLMs.

Problem

• part-level grounding
• visual grounding
• Multimodal Large Language Models
• point grounding
• fine-grained localization

Innovation

• part-level grounding
• point grounding
• multimodal large language models
• attention mechanism
• Q-Synth Module