LLM-wrapper: Black-Box Semantic-Aware Adaptation of Vision-Language Models for Referring Expression Comprehension

2024-09-18

AI Summary

This work addresses the practical challenge of limited zero-shot referring expression comprehension (REC) performance in closed-source vision-language models (VLMs), which cannot be white-box fine-tuned. We propose a black-box semantic adaptation framework that introduces a lightweight fine-tuned large language model (LLM) as a learnable semantic arbitrator—operating without access to the VLM’s internal parameters—to semantically align and re-rank candidate bounding boxes generated by the black-box VLM. Our method enables cross-model, cross-dataset, and multi-VLM integration while preserving privacy and enhancing generalization. It achieves significant zero-shot REC performance gains across multiple benchmarks, supports arbitrary VLM/LLM combinations, and demonstrates strong transferability. The code and models are publicly released.

Technology Category

• Computer Vision: Large Vision Models
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Vision Language Models (VLMs) have demonstrated remarkable capabilities in various open-vocabulary tasks, yet their zero-shot performance lags behind task-specific fine-tuned models, particularly in complex tasks like Referring Expression Comprehension (REC). Fine-tuning usually requires 'white-box' access to the model's architecture and weights, which is not always feasible due to proprietary or privacy concerns. In this work, we propose LLM-wrapper, a method for 'black-box' adaptation of VLMs for the REC task using Large Language Models (LLMs). LLM-wrapper capitalizes on the reasoning abilities of LLMs, improved with a light fine-tuning, to select the most relevant bounding box matching the referring expression, from candidates generated by a zero-shot black-box VLM. Our approach offers several advantages: it enables the adaptation of closed-source models without needing access to their internal workings, it is versatile as it works with any VLM, it transfers to new VLMs and datasets, and it allows for the adaptation of an ensemble of VLMs. We evaluate LLM-wrapper on multiple datasets using different VLMs and LLMs, demonstrating significant performance improvements and highlighting the versatility of our method. While LLM-wrapper is not meant to directly compete with standard white-box fine-tuning, it offers a practical and effective alternative for black-box VLM adaptation. Code and checkpoints are available at https://github.com/valeoai/LLM_wrapper .

Problem

• Improves zero-shot performance of Vision-Language Models in Referring Expression Comprehension.
• Enables black-box adaptation of VLMs without accessing internal model architecture.
• Utilizes LLMs for selecting relevant bounding boxes from VLM-generated candidates.

Innovation

• Black-box adaptation using LLMs for VLMs
• Light fine-tuning enhances LLM reasoning abilities
• Versatile method for closed-source VLM adaptation