Prompting-MammAlps: Fine-Grained Text-to-Video Retrieval for Camera-Trap Data

2026-07-10

AI Summary

This work addresses the limited spatiotemporal understanding and poor generalization of existing text-to-video retrieval methods on ecological camera trap data. To bridge this gap, the authors introduce Prompting-MammAlps, the first fine-grained text-to-video retrieval benchmark tailored for camera trap footage. Their approach leverages a vision transformer for action localization to generate structured video descriptions and incorporates ethology-inspired query templates alongside a hallucination-resistant, custom large language model (LLM) parsing library to enable interpretable and accurate retrieval. Evaluated on a test set comprising 135 ecological queries and 775 videos, the proposed method achieves a set-based F1 score of 34%, substantially outperforming zero-shot vision-language models (18%) while maintaining strong interpretability.

Abstract

Automatically retrieving videos from large camera-trap datasets remains challenging. Text-to-Video retrieval (TVR) methods based on large video-language models (VLMs) have potential to retrieve events of interest by describing them with simple text queries. However, current methods often lack spatiotemporal understanding and do not generalize well to ecological data. In this work, we introduce Prompting-MammAlps, the first camera-trap TVR benchmark, and propose a fine-grained and interpretable TVR method. Specifically, we trained a vision transformer to perform spatiotemporal action localization, and convert its output to structured text, describing each video. Independently, ethology-inspired queries are processed by a Large-Language Model (LLM) based coding agent to parse the structured text per video and retrieve videos accordingly. We harnessed the LLM to use functions from a custom parsing library to minimize the risk of LLM hallucinations and to improve method interpretability. This retrieval approach applied on the Prompting-MammAlps benchmark achieved a set-based F1-score of 34\% on a test set of 135 ecologically-relevant queries and 775 candidate videos. In comparison the best zero-shot VLM achieved a F1-score of 18\%, while also lacking interpretability. Project page: https://cnai.epfl.ch/prompting-mammalps

Problem

• Text-to-Video Retrieval
• Camera-Trap Data
• Spatiotemporal Understanding
• Ecological Data
• Fine-Grained Retrieval

Innovation

• text-to-video retrieval
• camera-trap data
• spatiotemporal action localization
• large language model
• structured text parsing