🤖 AI Summary
This work addresses the limited generalization of existing image classification models in cross-dataset and cross-domain scenarios. The authors propose a heterogeneous vision model ensemble system that leverages a multimodal large language model (MLLM) as a dynamic router to adaptively select the optimal backbone network—such as ResNet, self-supervised models, or vision-language models—at the sample level based on the input image. This approach enables seamless integration of new knowledge and unification of multi-source label spaces without requiring retraining. Evaluated across multiple heterogeneous datasets, the method achieves performance comparable to routers trained specifically for each task, while significantly enhancing cross-domain adaptability, interpretability, and scalability.
📝 Abstract
Modern image classification models excel when trained on single task-specific datasets but often struggle to generalize across domains and difficulty levels. We propose ARMDIL, an Adaptive Router for Multi-Domain Image classification with LLMs. ARMDIL is an ensemble that uses a multimodal large language model (MLLM) agent to dynamically route each image to the most suitable vision backbone. Our diverse ensemble employs convolutional neural networks (ResNets), self-supervised representation learners (SSL), and vision-language models (VLMs), each trained on a unified label space constructed from multiple image datasets with differing distributions and characteristics. Empirical evaluations illuminate the distinct capabilities and vulnerabilities of each architecture across disparate visual domains. Crucially, we show that ARMDIL effectively navigates these trade-offs, performing competitively with specialized training-based routers. Furthermore, it drastically improves adaptability by allowing new information to be integrated via simple prompt modifications, while enhancing interpretability through natural language reasoning traces. These advances in cross-dataset image classification pave the way for more reliable general-purpose vision systems such as AI assistants and autonomous robots.