DetRefiner: Model-Agnostic Detection Refinement with Feature Fusion Transformer

2026-05-11

AI Summary

Existing open-vocabulary object detection (OVOD) methods struggle to effectively integrate global and local contextual information, limiting their generalization to unseen categories. This work proposes a lightweight, plug-and-play refinement framework that operates solely on the outputs of a frozen base detector—without requiring access to or retraining of the underlying model. The framework employs a compact Transformer encoder to fuse global image features and local patch representations extracted from foundation models such as DINOv3, while simultaneously learning attribute reliability to calibrate detection confidence scores. Evaluated across multiple benchmarks—including COCO, LVIS, ODinW13, and Pascal VOC—the approach consistently enhances the performance of diverse OVOD models, achieving up to a +10.1 AP gain on unseen categories and demonstrating strong cross-model generalizability and novel-category recognition capability.

Abstract

Open-vocabulary object detection (OVOD) aims to detect both seen and unseen categories, yet existing methods often struggle to generalize to novel objects due to limited integration of global and local contextual cues. We propose DetRefiner, a simple yet effective plug-and-play framework that learns to fuse global and local features to refine open-vocabulary detection. DetRefiner processes global image features and patch-level image features from foundational models (e.g., DINOv3) through a lightweight Transformer encoder. The encoder produces a class vector capturing image-level attributes and patch vectors representing local region attributes, from which attribute reliability is inferred to recalibrate the base model's confidence. Notably, DetRefiner is trained independently of the base OVOD model, requiring neither access to its internal features nor retraining. At inference, it operates solely on the base detector's predictions, producing auxiliary calibration scores that are merged with the base detector's scores to yield the final refined confidence. Despite this simplicity, DetRefiner consistently enhances multiple OVOD models across COCO, LVIS, ODinW13, and Pascal VOC, achieving gains of up to +10.1 AP on novel categories. These results highlight that learning to fuse global and local representations offers a powerful and general mechanism for advancing open-world object detection. Our codes and models are available at https://github.com/hitachi-rd-cv/detrefiner.

Problem

• open-vocabulary object detection
• novel category generalization
• global-local context integration
• detection refinement

Innovation

• open-vocabulary object detection
• feature fusion
• Transformer
• model-agnostic refinement
• confidence calibration