Mind the Gap: Evaluating Vision Systems in Small Data Applications

2025-04-08

AI Summary

Current visual AI evaluation overemphasizes zero- or few-shot settings, neglecting realistic small-data regimes—such as ecological monitoring and medical diagnosis—where labeled samples number in the hundreds to thousands, annotation is costly, and practical solutions are urgently needed. Method: Leveraging the Natural World Tasks (NeWT) benchmark, this work conducts the first systematic comparison of multimodal large language models (MLLMs) against classical vision models (e.g., ViT, ResNet) under controlled, incrementally scaled training set sizes. Contribution/Results: MLLMs saturate in performance at ≤10 samples, whereas vision models exhibit consistent gains with increasing data; the performance gap widens substantially as sample size grows. These findings demonstrate that conventional vision architectures retain significant advantages in small-data regimes, underscoring the need for an explicit, standardized small-data evaluation paradigm—one that bridges theoretical AI advances with real-world applicability.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Large Vision Models
• Natural Language Processing: (Large) Language Models

Application Category

• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Search and Retrieval-Augmented AI: Large language models for search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

The practical application of AI tools for specific computer vision tasks relies on the "small-data regime" of hundreds to thousands of labeled samples. This small-data regime is vital for applications requiring expensive expert annotations, such as ecological monitoring, medical diagnostics or industrial quality control. We find, however, that computer vision research has ignored the small data regime as evaluations increasingly focus on zero- and few-shot learning. We use the Natural World Tasks (NeWT) benchmark to compare multi-modal large language models (MLLMs) and vision-only methods across varying training set sizes. MLLMs exhibit early performance plateaus, while vision-only methods improve throughout the small-data regime, with performance gaps widening beyond 10 training examples. We provide the first comprehensive comparison between these approaches in small-data contexts and advocate for explicit small-data evaluations in AI research to better bridge theoretical advances with practical deployments.

Problem

• Evaluating vision systems in small-data applications
• Comparing MLLMs and vision-only methods in small-data regimes
• Advocating for small-data evaluations to bridge theory and practice

Innovation

• Evaluates MLLMs and vision-only methods in small-data
• Uses Natural World Tasks benchmark for comparison
• Advocates explicit small-data evaluations in AI research