Mordal: Automated Pretrained Model Selection for Vision Language Models

2025-02-01

AI Summary

To address the inefficiency and manual dependency in visual-language model (VLM) selection, this paper proposes the first automated VLM model search framework. Our method introduces a task-aware model similarity metric, a lightweight surrogate evaluator, and a multi-stage candidate pruning strategy—jointly ensuring high evaluation fidelity while drastically reducing computational overhead. Compared to exhaustive grid search, our approach reduces GPU-hour consumption by 8.9–11.6×. It accurately identifies optimal VLM configurations across diverse downstream domains, achieving new state-of-the-art performance on multiple metrics. The core contribution is an end-to-end, efficient, and generalizable VLM selection paradigm that balances scalability with reproducibility, providing a systematic, deployable solution for practical VLM integration.

Technology Category

• Computer Vision: Large Vision Models
• Machine Learning: Large Multimodal Models (LMMs)
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Incorporating multiple modalities into large language models (LLMs) is a powerful way to enhance their understanding of non-textual data, enabling them to perform multimodal tasks. Vision language models (VLMs) form the fastest growing category of multimodal models because of their many practical use cases, including in healthcare, robotics, and accessibility. Unfortunately, even though different VLMs in the literature demonstrate impressive visual capabilities in different benchmarks, they are handcrafted by human experts; there is no automated framework to create task-specific multimodal models. We introduce Mordal, an automated multimodal model search framework that efficiently finds the best VLM for a user-defined task without manual intervention. Mordal achieves this both by reducing the number of candidates to consider during the search process and by minimizing the time required to evaluate each remaining candidate. Our evaluation shows that Mordal can find the best VLM for a given problem using up to $8.9 imes$--$11.6 imes$ lower GPU hours than grid search. In the process of our evaluation, we have also discovered new VLMs that outperform their state-of-the-art counterparts.

Problem

• Visual Language Models
• Automatic Model Selection
• Task-specific Optimization

Innovation

• Mordal System
• Automatic Model Selection
• Visual Language Models