Teach a Molmo2Fish: Towards interactive fish tracking with natural language guidance

2026-08-19

AI Summary

研究通过与多模态大语言模型的交互来修正鱼类追踪预测，提出Molmo2Fish工具，以提高生态学中计算机视觉任务的准确性。

Abstract

Computer vision is increasingly used to automate recognition tasks in large ecological datasets, but more complex tasks such as multi-object tracking continue to pose challenges. As researchers seek to incorporate vision models in ecology workflows, various lines of research have explored how to make imperfect predictions useful through human-in-the-loop processes. We propose a new approach to working with imperfect tracking predictions through an interactive prediction correction workflow taking place as a conversation with a multimodal large language model, which we tailor to a sonar fish tracking dataset as an initial proof of concept. We investigate the performance of the tool, Molmo2Fish, across guided and unguided tasks, correcting its own predicted tracks and external tracks. We find that Molmo2Fish achieves high performance on fish tracking and track correction tasks, but there is still much room to improve on incorporating natural language guidance. The code and data are publicly available at https://github.com/tidalove/molmo2fish.

Problem

• multi-object tracking
• natural language guidance
• imperfect predictions

Innovation

• interactive prediction correction
• multimodal large language model
• fish tracking
• natural language guidance