Detect Anything via Next Point Prediction

2025-10-14

AI Summary

Traditional coordinate-regression detectors (e.g., YOLO, DETR) exhibit limited generalization in zero-shot object detection, while existing multimodal large language models (MLLMs) suffer from low recall, duplicate predictions, and coordinate misalignment. To address these bottlenecks, we propose Rex-Omni—a 3B-parameter MLLM introducing the novel “next-coordinate-point prediction” paradigm for universal object detection. Our key contributions are: (1) quantized coordinate tokenization to simplify localization learning; (2) a multi-source data engine that synthesizes high-quality referring data; and (3) geometric-aware reward-guided reinforcement learning via GRPO for post-training, effectively mitigating duplication and misalignment. Rex-Omni supports diverse tasks—including zero-shot detection, pointing, image-text grounding, and OCR—without task-specific fine-tuning. On COCO and LVIS, it achieves zero-shot performance on par with or surpassing DINO, and demonstrates systematic superiority across multiple downstream benchmarks.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Computer Vision: Multi-modal Vision

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Object detection has long been dominated by traditional coordinate regression-based models, such as YOLO, DETR, and Grounding DINO. Although recent efforts have attempted to leverage MLLMs to tackle this task, they face challenges like low recall rate, duplicate predictions, coordinate misalignment, etc. In this work, we bridge this gap and propose Rex-Omni, a 3B-scale MLLM that achieves state-of-the-art object perception performance. On benchmarks like COCO and LVIS, Rex-Omni attains performance comparable to or exceeding regression-based models (e.g., DINO, Grounding DINO) in a zero-shot setting. This is enabled by three key designs: 1) Task Formulation: we use special tokens to represent quantized coordinates from 0 to 999, reducing the model's learning difficulty and improving token efficiency for coordinate prediction; 2) Data Engines: we construct multiple data engines to generate high-quality grounding, referring, and pointing data, providing semantically rich supervision for training; 3) Training Pipelines: we employ a two-stage training process, combining supervised fine-tuning on 22 million data with GRPO-based reinforcement post-training. This RL post-training leverages geometry-aware rewards to effectively bridge the discrete-to-continuous coordinate prediction gap, improve box accuracy, and mitigate undesirable behaviors like duplicate predictions that stem from the teacher-guided nature of the initial SFT stage. Beyond conventional detection, Rex-Omni's inherent language understanding enables versatile capabilities such as object referring, pointing, visual prompting, GUI grounding, spatial referring, OCR and key-pointing, all systematically evaluated on dedicated benchmarks. We believe that Rex-Omni paves the way for more versatile and language-aware visual perception systems.

Problem

• Improving object detection accuracy and recall rates for MLLMs
• Addressing coordinate misalignment and duplicate prediction issues
• Enabling versatile visual perception capabilities beyond conventional detection

Innovation

• Uses quantized coordinate tokens for efficient prediction
• Constructs multiple data engines for rich supervision
• Employs two-stage training with geometry-aware RL