A multi-stage augmented multimodal interaction network for fish feeding intensity quantification

2025-06-17

AI Summary

To address low accuracy in assessing fish feeding intensity, inefficient multimodal fusion, and unreliable decision-making in recirculating aquaculture systems, this paper proposes the Multi-stage Augmented Inter-modal Network (MAINet). MAINet introduces two novel components: (1) an Auxiliary-modality Reinforced Primary-modality Mechanism (ARPM) that enhances dominant modalities via auxiliary ones, and (2) an Evidence Reasoning (ER)-based decision fusion framework. It integrates image, audio, and water-wave signals using Channel Attention Feature Normalization (CAFN) and Dual-modality Attention Feature Normalization (DAFN). Experiments demonstrate that MAINet achieves consistent performance across accuracy, precision, recall, and F1-score—ranging from 96.76% to 96.79%—significantly outperforming unimodal, bimodal, and conventional multimodal fusion approaches. Ablation studies validate that ARPM and the ER framework critically improve model robustness and feature utilization efficiency.

Technology Category

• Machine Learning: Multimodal Learning
• Computer Vision: Multi-modal Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

In recirculating aquaculture systems, accurate and effective assessment of fish feeding intensity is crucial for reducing feed costs and calculating optimal feeding times. However, current studies have limitations in modality selection, feature extraction and fusion, and co-inference for decision making, which restrict further improvement in the accuracy, applicability and reliability of multimodal fusion models. To address this problem, this study proposes a Multi-stage Augmented Multimodal Interaction Network (MAINet) for quantifying fish feeding intensity. Firstly, a general feature extraction framework is proposed to efficiently extract feature information from input image, audio and water wave datas. Second, an Auxiliary-modality Reinforcement Primary-modality Mechanism (ARPM) is designed for inter-modal interaction and generate enhanced features, which consists of a Channel Attention Fusion Network (CAFN) and a Dual-mode Attention Fusion Network (DAFN). Finally, an Evidence Reasoning (ER) rule is introduced to fuse the output results of each modality and make decisions, thereby completing the quantification of fish feeding intensity. The experimental results show that the constructed MAINet reaches 96.76%, 96.78%, 96.79% and 96.79% in accuracy, precision, recall and F1-Score respectively, and its performance is significantly higher than the comparison models. Compared with models that adopt single-modality, dual-modality fusion and different decision-making fusion methods, it also has obvious advantages. Meanwhile, the ablation experiments further verified the key role of the proposed improvement strategy in improving the robustness and feature utilization efficiency of model, which can effectively improve the accuracy of the quantitative results of fish feeding intensity.

Problem

• Accurate fish feeding intensity assessment in aquaculture systems
• Improving multimodal fusion model accuracy and reliability
• Enhancing feature extraction and decision-making for feeding quantification

Innovation

• Multi-stage Augmented Multimodal Interaction Network
• Auxiliary-modality Reinforcement Primary-modality Mechanism
• Evidence Reasoning rule for decision fusion