Zero-shot Shark Tracking and Biometrics from Aerial Imagery

2025-01-10

AI Summary

To address the bottleneck of heavy reliance on labeled data and model training for shark identification in marine ecological monitoring, this paper proposes FLAIR—a zero-shot drone vision analytics framework that enables cross-species segmentation, tracking, and biometric extraction of sharks directly from aerial video without any labeling, training, or fine-tuning. FLAIR integrates SAM2’s video understanding capability with CLIP’s cross-modal semantic alignment, augmented by frame-level spatiotemporal consistency constraints and biologically inspired metrics (e.g., body length, tail-beat frequency). Evaluated on 18,000 nurse shark images, FLAIR achieves a Dice score of 0.81—surpassing supervised detection models and matching human-prompted SAM2 performance—while generalizing effectively to multiple shark species. This work establishes the first zero-shot, annotation-free, cross-species biometric analysis framework for sharks, introducing a novel paradigm for intelligent, in-situ ecological monitoring.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Vertical and domain-specific search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

The recent widespread adoption of drones for studying marine animals provides opportunities for deriving biological information from aerial imagery. The large scale of imagery data acquired from drones is well suited for machine learning (ML) analysis. Development of ML models for analyzing marine animal aerial imagery has followed the classical paradigm of training, testing, and deploying a new model for each dataset, requiring significant time, human effort, and ML expertise. We introduce Frame Level ALIgment and tRacking (FLAIR), which leverages the video understanding of Segment Anything Model 2 (SAM2) and the vision-language capabilities of Contrastive Language-Image Pre-training (CLIP). FLAIR takes a drone video as input and outputs segmentation masks of the species of interest across the video. Notably, FLAIR leverages a zero-shot approach, eliminating the need for labeled data, training a new model, or fine-tuning an existing model to generalize to other species. With a dataset of 18,000 drone images of Pacific nurse sharks, we trained state-of-the-art object detection models to compare against FLAIR. We show that FLAIR massively outperforms these object detectors and performs competitively against two human-in-the-loop methods for prompting SAM2, achieving a Dice score of 0.81. FLAIR readily generalizes to other shark species without additional human effort and can be combined with novel heuristics to automatically extract relevant information including length and tailbeat frequency. FLAIR has significant potential to accelerate aerial imagery analysis workflows, requiring markedly less human effort and expertise than traditional machine learning workflows, while achieving superior accuracy. By reducing the effort required for aerial imagery analysis, FLAIR allows scientists to spend more time interpreting results and deriving insights about marine ecosystems.

Problem

• Shark Counting
• Aerial Photography
• Unmanned Aerial Vehicles (UAVs)

Innovation

• FLAIR
• Autonomous Animal Identification
• Drone Videography