C-GAP: Class-Aware and Online Prompting Improves Vision-Language Models on Imbalanced Classes

2026-07-09

AI Summary

This work addresses the challenge of detecting rare categories under long-tailed distributions within a small label space. The authors propose C-GAP, a framework that enhances minority-class performance without requiring additional training or annotations, while keeping existing vision-language detectors frozen. C-GAP innovatively integrates class-aware composite image captions with large language model–driven online prompt optimization, dynamically refining prompts per image based on performance feedback. It further introduces a dynamic AP@0.5 thresholding mechanism for detection decisions. Evaluated on the COCO benchmark, the method boosts the minority-class AP@0.5 from 17.69 to 32.09—a relative improvement of 81%—and achieves up to a 53% gain in overall average precision for minority classes.

Abstract

Safety-critical perception systems must reliably detect rare object classes within small label spaces, a setting that long-tailed detection methods, designed for hundreds of classes with dense annotation, fundamentally do not address. Open-vocabulary detectors offer a promising alternative, as they use natural language queries at inference time, making prompt quality a first-class lever for detection performance. We exploit this property to address class imbalance: rather than retraining models or collecting additional annotations, we ask whether iteratively refining the language prompts, fed to frozen detectors, can improve minority class detection. We introduce C-GAP Caption-Guided Augmentation and Prompting), a detector-agnostic, annotation-free framework that operates in two phases. First, we establish a composite caption baseline combining per-image scene descriptions with class-quantity context, which we show outperforms scene-description only or class-quantity-only prompts across multiple open-vocabulary architectures and benchmarks. Second, an LLM iteratively refines each image's caption individually, with trials triaged into accept, tentative, or regenerate buckets based on minority-class AP@0.5 against a dynamic threshold derived from the composite baseline. Refinement terminates early once sufficient AP@0.5 gain is achieved. No detector weights are updated at any stage. Our experiments shows that C-GAP improves minority-class average precision up to 53% over the baselines. On COCO, C-GAP improves minority-class AP@0.5 by ~81% relative over the composite baseline (17.69 -> 32.09). Experiments confirm that composite captions provide the critical foundation for effective refinement: using scene-description-only or class-quantity-only prompts as the refinement starting point yields diminishing returns, supporting both stages of C-GAP as necessary contributions.

Problem

• long-tailed detection
• open-vocabulary detection
• class imbalance
• minority class detection
• vision-language models

Innovation

• open-vocabulary detection
• prompt engineering
• class imbalance
• vision-language models
• annotation-free refinement