🤖 AI Summary
The proliferation of models on platforms like Hugging Face has led users to rely heavily on popularity metrics—such as download counts and likes—for model selection, often overlooking actual performance. Method: This work systematically evaluates 500 sentiment analysis models, integrating ~80,000 human-annotated test instances, multi-dimensional evaluation (accuracy, robustness, bias), and audit of model card documentation. Contribution/Results: We empirically demonstrate that neither download count nor like count correlates significantly with true performance. Furthermore, 80% of model cards omit critical training or evaluation details, and 88% overstate performance. Based on these findings, we propose a user-oriented, actionable model selection checklist; quantify the prevalence of documentation deficiencies; and rigorously establish that popularity metrics are invalid proxies for model capability—thereby providing both methodological grounding and practical guidance for trustworthy model reuse.
📝 Abstract
With the massive surge in ML models on platforms like Hugging Face, users often lose track and struggle to choose the best model for their downstream tasks, frequently relying on model popularity indicated by download counts, likes, or recency. We investigate whether this popularity aligns with actual model performance and how the comprehensiveness of model documentation correlates with both popularity and performance. In our study, we evaluated a comprehensive set of 500 Sentiment Analysis models on Hugging Face. This evaluation involved massive annotation efforts, with human annotators completing nearly 80,000 annotations, alongside extensive model training and evaluation. Our findings reveal that model popularity does not necessarily correlate with performance. Additionally, we identify critical inconsistencies in model card reporting: approximately 80% of the models analyzed lack detailed information about the model, training, and evaluation processes. Furthermore, about 88% of model authors overstate their models' performance in the model cards. Based on our findings, we provide a checklist of guidelines for users to choose good models for downstream tasks.