YOLOE: Real-Time Seeing Anything

2025-03-10

AI Summary

To address the limitations of conventional detection/segmentation models (e.g., YOLO) in open-world scenarios—namely, their reliance on predefined categories—and the trade-off between performance and efficiency in existing open-vocabulary approaches, this paper proposes a unified, efficient single-stage framework. Built upon YOLOv8, it introduces the first general-purpose architecture integrating three paradigms: text prompting, visual prompting, and prompt-free inference. We propose three core techniques: RepRTA (reparameterizable region-text alignment), SAVPE (semantic-activated visual prompt encoding), and LRPC (lazy region-wise contrastive learning), jointly balancing zero-shot generalization, inference speed, and training efficiency. On LVIS, our method surpasses YOLO-Worldv2 with one-third the training cost and 1.4× faster inference. When transferred to COCO, it achieves higher mAP while reducing training time by nearly 4×.

Technology Category

• Computer Vision: Large Vision Models
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Prompt Engineering / Prompting

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Object detection and segmentation are widely employed in computer vision applications, yet conventional models like YOLO series, while efficient and accurate, are limited by predefined categories, hindering adaptability in open scenarios. Recent open-set methods leverage text prompts, visual cues, or prompt-free paradigm to overcome this, but often compromise between performance and efficiency due to high computational demands or deployment complexity. In this work, we introduce YOLOE, which integrates detection and segmentation across diverse open prompt mechanisms within a single highly efficient model, achieving real-time seeing anything. For text prompts, we propose Re-parameterizable Region-Text Alignment (RepRTA) strategy. It refines pretrained textual embeddings via a re-parameterizable lightweight auxiliary network and enhances visual-textual alignment with zero inference and transferring overhead. For visual prompts, we present Semantic-Activated Visual Prompt Encoder (SAVPE). It employs decoupled semantic and activation branches to bring improved visual embedding and accuracy with minimal complexity. For prompt-free scenario, we introduce Lazy Region-Prompt Contrast (LRPC) strategy. It utilizes a built-in large vocabulary and specialized embedding to identify all objects, avoiding costly language model dependency. Extensive experiments show YOLOE's exceptional zero-shot performance and transferability with high inference efficiency and low training cost. Notably, on LVIS, with 3$ imes$ less training cost and 1.4$ imes$ inference speedup, YOLOE-v8-S surpasses YOLO-Worldv2-S by 3.5 AP. When transferring to COCO, YOLOE-v8-L achieves 0.6 AP$^b$ and 0.4 AP$^m$ gains over closed-set YOLOv8-L with nearly 4$ imes$ less training time. Code and models are available at https://github.com/THU-MIG/yoloe.

Problem

• Overcoming predefined category limitations in object detection and segmentation.
• Balancing performance and efficiency in open-set object detection methods.
• Achieving real-time adaptability across diverse open prompt mechanisms.

Innovation

• Re-parameterizable Region-Text Alignment strategy
• Semantic-Activated Visual Prompt Encoder
• Lazy Region-Prompt Contrast strategy