GEAR-Seg: A Grounded Explainable Agent for Reasoning Segmentation and Data Engine

2026-07-01

AI Summary

This work addresses the poor interpretability and limited scalability of end-to-end reasoning-based segmentation models, which stem from the tight coupling of perception and reasoning. To resolve this, we propose the first explicit decoupling framework that transforms pixel inputs into traceable textual reasoning chains through a three-stage pipeline: class-agnostic segmentation, visual semantic caption generation, and logical reasoning via a large language model (LLM). This approach yields an interpretable and scalable zero-shot reasoning system, complemented by an automated data distillation engine. Our method achieves state-of-the-art performance across multiple benchmarks for reasoning and fine-grained referring expression segmentation. We also introduce the GEAR-131K dataset, comprising 38K images and 656K question-mask pairs, and demonstrate that lightweight models trained via our automated distillation approach closely approach the performance ceiling established by human annotations.

Abstract

Reasoning segmentation requires localizing targets based on complex, implicit queries. Current end-to-end models typically entangle perception and deduction into an opaque black box, severely limiting interpretability and scalability. To address this, we propose GEAR-Seg (Grounded Explainable Agent for Reasoning Segmentation), an explicitly decoupled agent that shifts the paradigm by translating visual pixels into dense, attribute-rich text. By decoupling class-agnostic segmentation, semantic description, and Large Language Model (LLM) deduction, GEAR-Seg transforms implicit reasoning into an explicit, trackable logic chain. As a zero-shot inference framework, it achieves highly competitive performance across diverse reasoning and fine-grained referring segmentation benchmarks. Furthermore, GEAR-Seg inherently functions as a highly scalable data engine. Utilizing this engine, we construct GEAR-131K, a massive benchmark (over 38k images, 656k QA-mask pairs) introducing a multifaceted taxonomy tailored for complex real-world manipulation-oriented reasoning. Finally, distillation experiments demonstrate that lightweight models supervised exclusively by our automated pipeline closely match the upper-bound performance of costly human-annotated baselines.

Problem

• reasoning segmentation
• interpretability
• scalability
• implicit queries
• black-box models

Innovation

• reasoning segmentation
• explainable AI
• decoupled architecture
• data engine
• zero-shot inference