EvoVLMA: Evolutionary Vision-Language Model Adaptation

2025-08-02

AI Summary

Existing vision-language model (VLM) adaptation methods—such as prompt tuning and adapter-based approaches—rely on manual design, suffering from low efficiency and lacking automated architecture search mechanisms. Method: This paper proposes the first automated algorithm design framework for zero-training VLM adaptation, featuring a two-stage LLM-assisted evolutionary algorithm that jointly optimizes feature selection and logits computation structure. To enable scalable exploration of large search spaces, it incorporates low-precision code translation and web-based execution monitoring. Contribution/Results: It is the first work to apply evolutionary algorithms to automatically construct VLM adaptation pipelines. Evaluated on 8-shot image classification, the discovered algorithm outperforms the manually designed APE method by +1.91 percentage points, demonstrating both effectiveness and state-of-the-art performance.

Technology Category

• Computer Vision: Large Vision Models
• Machine Learning: Large Multimodal Models (LMMs)
• Search and Optimization: Learning to Search

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

Pre-trained Vision-Language Models (VLMs) have been exploited in various Computer Vision tasks (e.g., few-shot recognition) via model adaptation, such as prompt tuning and adapters. However, existing adaptation methods are designed by human experts, requiring significant time cost and experience. Inspired by recent advances in Large Language Models (LLMs) based code generation, we propose an Evolutionary Vision-Language Model Adaptation (EvoVLMA) method to automatically search training-free efficient adaptation algorithms for VLMs. We recognize feature selection and logits computation as the key functions in training-free VLM adaptation, and propose a two-stage LLM-assisted evolutionary algorithm for optimizing these parts in a sequential manner, effectively addressing the challenge posed by the expansive search space through a divide-and-conquer strategy. Besides, to enhance the stability and efficiency of searching process, we propose low-precision code conversion, web based code execution and process monitoring, leading to a highly effective automatic algorithm design system. Extensive experiments demonstrate that the algorithms found by EvoVLMA can obtain promising results compared to previous manually-designed ones. More specifically, in the 8-shot image classification setting, the classical APE algorithm can be improved by 1.91 points in recognition accuracy. This research opens new possibilities for automating the optimization of adaptation algorithms of pre-trained multimodal models. Code is available at: https://github.com/kding1225/EvoVLMA

Problem

• Automate search for training-free VLM adaptation algorithms
• Optimize feature selection and logits computation via evolution
• Enhance search stability with low-precision code conversion

Innovation

• LLM-assisted evolutionary algorithm for VLM adaptation
• Two-stage optimization for feature and logits
• Low-precision code conversion for efficient search