🤖 AI Summary
This work addresses two key challenges in few-shot visual detection (<1,000 images): the strong data dependency of CNNs and the low fine-tuning efficiency of multimodal large language models (MLLMs). We propose a specialized fine-tuning paradigm for text-annotated detection tasks, explicitly modeling the synergy between language guidance and visual localization. Our approach integrates prompt engineering, dynamic context reasoning, and lightweight end-to-end fine-tuning. Under extreme data scarcity, the method achieves a 36% mAP improvement over conventional CNN baselines—matching or even surpassing their fully supervised performance. To our knowledge, this is the first work to empirically demonstrate the strong generalization capability of MLLMs in domain-specific few-shot visual detection. It establishes a novel cross-modal few-shot learning paradigm grounded in task-aware textual supervision. The implementation is publicly available.
📝 Abstract
The field of object detection and understanding is rapidly evolving, driven by advances in both traditional CNN-based models and emerging multi-modal large language models (LLMs). While CNNs like ResNet and YOLO remain highly effective for image-based tasks, recent transformer-based LLMs introduce new capabilities such as dynamic context reasoning, language-guided prompts, and holistic scene understanding. However, when used out-of-the-box, the full potential of LLMs remains underexploited, often resulting in suboptimal performance on specialized visual tasks. In this work, we conduct a comprehensive comparison of fine-tuned traditional CNNs, zero-shot pre-trained multi-modal LLMs, and fine-tuned multi-modal LLMs on the challenging task of artificial text overlay detection in images. A key contribution of our study is demonstrating that LLMs can be effectively fine-tuned on very limited data (fewer than 1,000 images) to achieve up to 36% accuracy improvement, matching or surpassing CNN-based baselines that typically require orders of magnitude more data. By exploring how language-guided models can be adapted for precise visual understanding with minimal supervision, our work contributes to the broader effort of bridging vision and language, offering novel insights into efficient cross-modal learning strategies. These findings highlight the adaptability and data efficiency of LLM-based approaches for real-world object detection tasks and provide actionable guidance for applying multi-modal transformers in low-resource visual environments. To support continued progress in this area, we have made the code used to fine-tune the models available in our GitHub, enabling future improvements and reuse in related applications.