An interpretable framework using foundation models for fish sex identification

2026-02-21

AI Summary

This study addresses the limitations of traditional invasive methods for fish sex identification, which pose stress and survival risks to endangered species such as the Delta Smelt (*Hypomesus transpacificus*). To overcome this challenge, the authors propose FishProtoNet, a novel framework that integrates vision foundation models with interpretable prototype learning. The vision foundation model automatically extracts regions of interest from fish images and generates deep features, while the prototype learning component enables transparent, non-invasive sex classification. This approach significantly enhances model robustness and interpretability, making it suitable for monitoring endangered fish throughout their life cycle. Evaluated on Delta Smelt during pre-spawning and post-spawning stages, the method achieves sex identification accuracies of 74.40% and 81.16%, with F1 scores of 74.27% and 79.43%, respectively.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding

Application Category

• Security and Privacy: Data transparency and provenance
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Accurate sex identification in fish is vital for optimizing breeding and management strategies in aquaculture, particularly for species at the risk of extinction. However, most existing methods are invasive or stressful and may cause additional mortality, posing severe risks to threatened or endangered fish populations. To address these challenges, we propose FishProtoNet, a robust, non-invasive computer vision-based framework for sex identification of delta smelt (Hypomesus transpacificus), an endangered fish species native to California, across its full life cycle. Unlike the traditional deep learning methods, FishProtoNet provides interpretability through learned prototype representations while improving robustness by leveraging foundation models to reduce the influence of background noise. Specifically, the FishProtoNet framework consists of three key components: fish regions of interest (ROIs) extraction using visual foundation model, feature extraction from fish ROIs and fish sex identification based on an interpretable prototype network. FishProtoNet demonstrates strong performance in delta smelt sex identification during early spawning and post-spawning stages, achieving the accuracies of 74.40% and 81.16% and corresponding F1 scores of 74.27% and 79.43% respectively. In contrast, delta smelt sex identification at the subadult stage remains challenging for current computer vision methods, likely due to less pronounced morphological differences in immature fish. The source code of FishProtoNet is publicly available at: https://github.com/zhengmiao1/Fish_sex_identification

Problem

• fish sex identification
• non-invasive
• endangered species
• aquaculture
• morphological differences

Innovation

• foundation models
• interpretable AI
• prototype learning
• non-invasive sex identification
• computer vision