Is AI currently capable of identifying wild oysters? A comparison of human annotators against the AI model, ODYSSEE
🤖 AI Summary
This study addresses the bottleneck in oyster reef ecological monitoring—reliance on destructive sampling and manual interpretation—by evaluating the ODYSSEE deep learning model for live oyster detection in field-collected video and image data. A customized YOLO-based object detection model was trained and tested on a real-world oyster reef dataset. For the first time, its performance was systematically benchmarked against both domain experts and non-expert annotators. Results reveal a model tendency toward over-prediction and an inverse relationship between image quality and model accuracy: higher-quality images led to lower detection accuracy, contrary to human annotators. Although inference speed vastly exceeds manual annotation (39.6 seconds versus several hours), the model’s live-oyster classification accuracy (63%) remains below that of experts (74%) and non-experts (75%). The study highlights fundamental limitations of AI in complex field settings and proposes novel pathways for human-AI collaborative optimization in ecological monitoring.
Application Category
📝 Abstract
Oysters are ecologically and commercially important species that require frequent monitoring to track population demographics (e.g. abundance, growth, mortality). Current methods of monitoring oyster reefs often require destructive sampling methods and extensive manual effort. Therefore, they are suboptimal for small-scale or sensitive environments. A recent alternative, the ODYSSEE model, was developed to use deep learning techniques to identify live oysters using video or images taken in the field of oyster reefs to assess abundance. The validity of this model in identifying live oysters on a reef was compared to expert and non-expert annotators. In addition, we identified potential sources of prediction error. Although the model can make inferences significantly faster than expert and non-expert annotators (39.6 s, $2.34 pm 0.61$ h, $4.50 pm 1.46$ h, respectively), the model overpredicted the number of live oysters, achieving lower accuracy (63%) in identifying live oysters compared to experts (74%) and non-experts (75%) alike. Image quality was an important factor in determining the accuracy of the model and the annotators. Better quality images improved human accuracy and worsened model accuracy. Although ODYSSEE was not sufficiently accurate, we anticipate that future training on higher-quality images, utilizing additional live imagery, and incorporating additional annotation training classes will greatly improve the model's predictive power based on the results of this analysis. Future research should address methods that improve the detection of living vs. dead oysters.
Problem
Research questions and friction points this paper is trying to address.
Evaluating AI's ability to identify wild oysters accurately
Comparing ODYSSEE model performance with human annotators
Identifying factors affecting oyster detection accuracy
Innovation
Methods, ideas, or system contributions that make the work stand out.
Uses deep learning for oyster identification
Compares AI accuracy with human annotators
Improves model with high-quality images
🔎 Similar Papers
No similar papers found.
