Segmentation before Answering: Pixel Grounding for MLLM Visual Reasoning

2026-07-06

AI Summary

Current multimodal large language models (MLLMs) rely on coarse-grained bounding boxes for region-focused visual reasoning, which are prone to background interference and misaligned with the underlying visual token structure. This work proposes SegAnswer, the first approach to integrate pixel-level segmentation masks into the MLLM inference pipeline, replacing bounding boxes with fine-grained masks as the visual focus units. SegAnswer leverages a segmentation model to generate masks, crops the input image accordingly, and fuses positional embeddings to achieve precise region isolation and natural alignment with visual tokens. Experiments demonstrate consistent improvements across tasks requiring high-resolution perception, general visual understanding, and hallucination suppression, while also exhibiting robust pixel-level localization capabilities.

Abstract

Recent advancements in Multimodal Large Language Models (MLLMs) have evolved from static perception to interleaved visual-language reasoning, often referred to as ``thinking with images''. A basic operation in this reasoning process is to zoom in on regions of interest (often represented with bounding boxes) to acquire finer visual details. In this paper, we propose \textbf{Seg}mentation before \textbf{Answer}ing (SegAnswer), which shifts the unit of zoom-in from the popular bounding box to pixel-level segmentation mask. By employing fine-grained masks to isolate the target area from cluttered environments, segmented visual input yields a more precise region of interest, effectively filtering out redundant background and interfering objects. Furthermore, the discrete patches of segmented visual input align more seamlessly with how MLLMs structure visual tokens via positional embeddings. In experiments, we evaluate SegAnswer across diverse benchmarks, including high-resolution perception, general perception, and hallucination. It achieves consistent improvements and also exhibits considerable performance on segmentation tasks, validating its capability for reliable pixel grounding.

Problem

• Multimodal Large Language Models
• visual reasoning
• pixel grounding
• region of interest
• segmentation

Innovation

• pixel grounding
• segmentation mask
• multimodal large language models
• visual reasoning
• region of interest