Fish-Vista: A Multi-Purpose Dataset for Understanding & Identification of Traits from Images

2024-07-10

arXiv.org

AI Summary

High-quality benchmark data for visual trait analysis of aquatic organisms remains scarce. Method: We introduce Fish-Vista, the first AI-ready multi-task fish image dataset, comprising 69,126 expert-verified images spanning 4,154 fish species, supporting species classification, trait recognition, and trait segmentation. We propose a reproducible cross-collection image cleaning and alignment pipeline, integrating labels from multiple biological databases and incorporating domain-expert validation; we further design a unified computer vision benchmark framework for systematic evaluation of state-of-the-art models. Contribution/Results: Fish-Vista establishes novel paradigms for long-tail learning, weakly supervised segmentation, and explainability assessment in ecological vision analysis, uncovering critical challenges including out-of-distribution generalization and small-object segmentation. As the largest publicly available fish visual trait benchmark to date, Fish-Vista provides an open infrastructure to advance AI-driven biodiversity science.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multi-instance/Multi-view Learning
• Humans and AI: Crowd Sourcing and Human Computation

Application Category

• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Security and Privacy: Data transparency and provenance
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

We introduce Fish-Visual Trait Analysis (Fish-Vista), the first organismal image dataset designed for the analysis of visual traits of aquatic species directly from images using problem formulations in computer vision. Fish-Vista contains 69,126 annotated images spanning 4,154 fish species, curated and organized to serve three downstream tasks of species classification, trait identification, and trait segmentation. Our work makes two key contributions. First, we perform a fully reproducible data processing pipeline to process images sourced from various museum collections. We annotate these images with carefully curated labels from biological databases and manual annotations to create an AI-ready dataset of visual traits, contributing to the advancement of AI in biodiversity science. Second, our proposed downstream tasks offer fertile grounds for novel computer vision research in addressing a variety of challenges such as long-tailed distributions, out-of-distribution generalization, learning with weak labels, explainable AI, and segmenting small objects. We benchmark the performance of several existing methods for our proposed tasks to expose future research opportunities in AI for biodiversity science problems involving visual traits.

Problem

• Dataset for aquatic species trait analysis
• Species classification and trait identification
• Challenges in biodiversity AI research

Innovation

• Reproducible image processing pipeline
• AI-ready dataset with curated labels
• Benchmarking for biodiversity science tasks