Towards Pixel-Level VLM Perception via Simple Points Prediction

2026-01-27

AI Summary

This work proposes a novel approach to endow multimodal large language models (MLLMs) with native pixel-level perception capabilities without requiring specialized architectures. By reformulating image segmentation as a sequence generation task of coordinate points in the language space, the method leverages standard MLLM architectures and employs a two-stage training strategy: supervised fine-tuning (SFT) followed by reinforcement learning (RL) optimization using Intersection over Union (IoU) as the reward signal. This approach significantly enhances segmentation accuracy and demonstrates, for the first time, that general-purpose MLLMs inherently possess strong low-level visual perception potential. Remarkably, using only simple point prediction, the model achieves competitive or superior performance against existing complex, task-specific architectures across multiple segmentation benchmarks, offering both efficiency and broad applicability.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Large Vision Models
• Natural Language Processing: (Large) Language Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

We present SimpleSeg, a strikingly simple yet highly effective approach to endow Multimodal Large Language Models (MLLMs) with native pixel-level perception. Our method reframes segmentation as a simple sequence generation problem: the model directly predicts sequences of points (textual coordinates) delineating object boundaries, entirely within its language space. To achieve high fidelity, we introduce a two-stage SF$\to$RL training pipeline, where Reinforcement Learning with an IoU-based reward refines the point sequences to accurately match ground-truth contours. We find that the standard MLLM architecture possesses a strong, inherent capacity for low-level perception that can be unlocked without any specialized architecture. On segmentation benchmarks, SimpleSeg achieves performance that is comparable to, and often surpasses, methods relying on complex, task-specific designs. This work lays out that precise spatial understanding can emerge from simple point prediction, challenging the prevailing need for auxiliary components and paving the way for more unified and capable VLMs. Homepage: https://simpleseg.github.io/

Problem

• pixel-level perception
• Multimodal Large Language Models
• image segmentation
• spatial understanding
• vision-language models

Innovation

• pixel-level perception
• point sequence prediction
• multimodal large language models
• reinforcement learning
• segmentation