CollagePrompt: A Benchmark for Budget-Friendly Visual Recognition with GPT-4V

2024-03-18

AI Summary

GPT-4V achieves strong visual recognition performance but incurs prohibitively high inference costs. To address this, we propose the “collage prompting” paradigm—synthesizing multiple images into a single composite input to enable batched inference and substantially reduce cost. We introduce CollagePrompt, the first dedicated benchmark for collage prompting, comprising diverse collage datasets and a standardized evaluation protocol. We systematically identify and quantify three critical factors affecting accuracy: positional bias, class clustering gain, and adjacency interference. Furthermore, we propose a genetic algorithm–based automatic layout optimization method and define two novel efficiency metrics. Experiments demonstrate that our approach reduces GPT-4V invocation costs by over 50% across multiple vision tasks while preserving competitive accuracy. All code and datasets are publicly released to advance research on cost-efficient multimodal visual understanding.

Technology Category

• Computer Vision: Large Vision Models
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Prompt Engineering / Prompting

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG

Abstract

Recent advancements in generative AI have suggested that by taking visual prompts, GPT-4V can demonstrate significant proficiency in visual recognition tasks. Despite its impressive capabilities, the financial cost associated with GPT-4V's inference presents a substantial barrier to its wide use. To address this challenge, we propose a budget-friendly collage prompting task that collages multiple images into a single visual prompt and makes GPT-4V perform visual recognition on several images simultaneously, thereby reducing the cost. We collect a dataset of various collage prompts to assess its performance in GPT-4V's visual recognition. Our evaluations reveal several key findings: 1) Recognition accuracy varies with different positions in the collage. 2) Grouping images of the same category together leads to better visual recognition results. 3) Incorrect labels often come from adjacent images. These findings highlight the importance of image arrangement within collage prompt. To this end, we construct a benchmark called CollagePrompt, which offers a platform for designing collage prompt to achieve more cost-effective visual recognition with GPT-4V. A baseline method derived from genetic algorithms to optimize collage layouts is proposed and two metrics are introduced to measure the efficiency of the optimized collage prompt. Our benchmark enables researchers to better optimize collage prompts, thus making GPT-4V more cost-effective in visual recognition. The code and data are available at this project page https://collageprompting.github.io/.

Problem

• Reduces GPT-4V visual recognition costs.
• Optimizes collage layouts for better accuracy.
• Introduces CollagePrompt benchmark for efficiency.

Innovation

• Collage prompting reduces GPT-4V costs.
• Genetic algorithms optimize collage layouts.
• Grouping same-category images enhances accuracy.