CleverBirds: A Multiple-Choice Benchmark for Fine-grained Human Knowledge Tracing

2025-11-11

AI Summary

This study aims to precisely model the evolution of human knowledge in fine-grained bird identification. Method: We introduce BirdKT, the first large-scale knowledge tracing benchmark designed for visual expert skill development, built from 17 million multiple-choice questions submitted by 400,000 eBird users—covering over 10,000 bird species and long-term learning sequences. Our approach models multiple-choice response sequences by integrating contextual cues—including species similarity and geographic distribution—to support both fine-grained classification and longitudinal learning pattern analysis. Contribution/Results: Empirical analysis reveals that users answer an average of 400 items, with substantial inter-individual trajectory heterogeneity; existing knowledge tracing models exhibit limited cross-context generalization. BirdKT establishes a novel, scalable benchmark for visual learning modeling, introduces new challenges in long-horizon visual knowledge tracking, and provides an extensible evaluation platform for future research.

Technology Category

• Data Mining & Knowledge Management: Mining of Visual, Multimedia & Multimodal Data
• Knowledge Representation and Reasoning: Knowledge Acquisition
• Machine Learning: Learning Preferences or Rankings

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Semantics and Knowledge: Scalable techniques for the creation, curation, publication, maintenance, and consumption of large, Web-based, structured, reusable, knowledge graphs and ontologies
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Mastering fine-grained visual recognition, essential in many expert domains, can require that specialists undergo years of dedicated training. Modeling the progression of such expertize in humans remains challenging, and accurately inferring a human learner's knowledge state is a key step toward understanding visual learning. We introduce CleverBirds, a large-scale knowledge tracing benchmark for fine-grained bird species recognition. Collected by the citizen-science platform eBird, it offers insight into how individuals acquire expertize in complex fine-grained classification. More than 40,000 participants have engaged in the quiz, answering over 17 million multiple-choice questions spanning over 10,000 bird species, with long-range learning patterns across an average of 400 questions per participant. We release this dataset to support the development and evaluation of new methods for visual knowledge tracing. We show that tracking learners'knowledge is challenging, especially across participant subgroups and question types, with different forms of contextual information offering varying degrees of predictive benefit. CleverBirds is among the largest benchmark of its kind, offering a substantially higher number of learnable concepts. With it, we hope to enable new avenues for studying the development of visual expertize over time and across individuals.

Problem

• Modeling human expertise progression in fine-grained visual recognition domains
• Inferring learners' knowledge states for understanding visual learning processes
• Developing methods to track knowledge across diverse participants and question types

Innovation

• Large-scale benchmark for fine-grained bird species recognition
• Dataset from citizen-science platform with long-range learning patterns
• Supports knowledge tracing across participant subgroups and question types