Few-Class Arena: A Benchmark for Efficient Selection of Vision Models and Dataset Difficulty Measurement

2024-11-02

arXiv.org

AI Summary

Existing vision models are predominantly evaluated on thousand-class benchmarks, yet real-world deployments often require only 2–10 classes. This mismatch impedes reliable prediction of few-class performance from multi-class metrics and lacks dedicated evaluation protocols. Method: We introduce Few-Class Arena (FCA), the first unified benchmark and toolchain tailored to practical few-class scenarios (2–10 classes). Contribution/Results: FCA establishes (1) a standardized evaluation framework for the few-class paradigm; (2) a dataset difficulty quantification function based on inter-class semantic similarity; and (3) lightweight model design principles and a novel scaling law. Comprehensive evaluation across ImageNet subsets (2–1000 classes) demonstrates that few-class accuracy cannot be extrapolated from multi-class metrics. FCA is open-sourced, supporting 10 datasets and plug-and-play testing across diverse architectures, thereby significantly improving model-task alignment efficiency.

Technology Category

• Computer Vision: Large Vision Models
• Machine Learning: Calibration & Uncertainty Quantification
• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success
• Security and Privacy: Large-scale security measurements

Abstract

We propose Few-Class Arena (FCA), as a unified benchmark with focus on testing efficient image classification models for few classes. A wide variety of benchmark datasets with many classes (80-1000) have been created to assist Computer Vision architectural evolution. An increasing number of vision models are evaluated with these many-class datasets. However, real-world applications often involve substantially fewer classes of interest (2-10). This gap between many and few classes makes it difficult to predict performance of the few-class applications using models trained on the available many-class datasets. To date, little has been offered to evaluate models in this Few-Class Regime. We conduct a systematic evaluation of the ResNet family trained on ImageNet subsets from 2 to 1000 classes, and test a wide spectrum of Convolutional Neural Networks and Transformer architectures over ten datasets by using our newly proposed FCA tool. Furthermore, to aid an up-front assessment of dataset difficulty and a more efficient selection of models, we incorporate a difficulty measure as a function of class similarity. FCA offers a new tool for efficient machine learning in the Few-Class Regime, with goals ranging from a new efficient class similarity proposal, to lightweight model architecture design, to a new scaling law. FCA is user-friendly and can be easily extended to new models and datasets, facilitating future research work. Our benchmark is available at https://github.com/bryanbocao/fca.

Problem

• Evaluates vision models for few-class image classification.
• Measures dataset difficulty based on class similarity.
• Provides a benchmark for efficient model selection.

Innovation

• Few-Class Arena benchmark for few-class image classification
• Evaluates ResNet, CNNs, Transformers on 2-1000 class subsets
• Incorporates class similarity for dataset difficulty assessment