Image Embedding Sampling Method for Diverse Captioning

2025-02-14

AI Summary

To address the limited caption diversity, insufficient fine-grained detail, and poor deployability of lightweight vision-language models (e.g., BLIP) on resource-constrained devices, this paper proposes a training-free embedding-space region sampling framework. Our method leverages hierarchical image segmentation (e.g., SAM) to construct global-local semantic representations, and employs multi-granularity embedding aggregation coupled with semantics-guided structured sampling to explicitly model fine-grained visual content during zero-shot inference. Crucially, it enables high-quality caption generation without parameter expansion or fine-tuning. Experiments demonstrate state-of-the-art diversity (Div-2 scores of 0.735, 0.750, and 0.748 on MSCOCO, Flickr30k, and NoCaps, respectively), alongside significantly improved image–text alignment and higher consistency with human annotations compared to same-scale baselines.

Technology Category

• Computer Vision: Large Vision Models
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Image Captioning for state-of-the-art VLMs has significantly improved over time; however, this comes at the cost of increased computational complexity, making them less accessible for resource-constrained applications such as mobile devices and assistive technologies. Alternatively, smaller VLMs prioritize high-level scene descriptions, overlooking finer details that contribute to a richer understanding of an image. In this paper, we introduce a training-free framework that enhances caption diversity and informativeness by explicitly attending to distinct image regions using a comparably small VLM, BLIP, as the backbone. Our approach leverages structured segmentation to produce hierarchical representations that capture both global and localized semantics. Without requiring additional model training, we demonstrate that our method allows smaller VLMs to achieve performance comparable to larger models in terms of image-caption alignment, semantic integrity, and diversity. We evaluate our framework on MSCOCO, Flickr30k, and Nocaps test datasets, achieving a Div-2 score of 0.735, 0.750, and 0.748 for each dataset respectively, while maintaining strong image-caption relevancy and semantic integrity with the human-annotated captions.

Problem

• Enhances caption diversity and informativeness
• Reduces computational complexity for VLMs
• Improves performance of smaller VLMs

Innovation

• Training-free framework
• Hierarchical representations
• Small VLM performance