K9-Bench: Evaluating Multimodal LLMs on Canine-Centric Videos

2026-07-02

AI Summary

This work addresses the lack of systematic evaluation of large-scale multimodal language models (MLLMs) on pet—particularly domestic dog—video understanding, which hinders practical applications such as canine emotion recognition, behavior analysis, and long-term interaction. To bridge this gap, we introduce K9-Bench, the first dog-centric multimodal benchmark comprising 907 real-world videos and approximately 5,000 question-answer pairs spanning five fine-grained tasks. We propose a scalable data generation pipeline that synergistically combines vision-language models (VLMs) and large language models (LLMs), incorporating multi-hop reasoning, temporal modeling, and bias mitigation strategies tailored for low-resource animal behavior domains. Experimental results reveal that state-of-the-art models exhibit limited zero-shot performance on dog-related tasks, especially in long-horizon pose and interaction reasoning, with standard chain-of-thought prompting yielding only marginal improvements.

Abstract

MLLMs have shown strong zero-shot capabilities across diverse inputs such as across images, video, audio, and text. A crucial, yet underexplored, application of these models lies in understanding and modeling animal-centric scenarios. As animals are integral to millions of households, benchmarking next-generation AI models on pet-focused tasks, ranging from recognizing distress signals to enabling responsive robotic companions, is essential for building AI systems that can work alongside us. We introduce K9-Bench, a novel benchmark focused on real-world domestic dog videos, specifically targeting canine action and interaction understanding via approximately 5000 question-answer pairs across 907 videos spanning 5 distinct task categories that test long-form, canine-centric multimodal reasoning in MLLMs. To create this dataset, we propose a scalable, VLM/LLM-powered data generation pipeline that automatically mines canine-centric videos from the web and curates QA pairs requiring fine-grained, multi-hop reasoning over canine actions and temporally extended interaction sequences. We implement bias mitigation strategies designed to eliminate biases introduced by VLMs during dataset curation. Through extensive experimentation, we find that frontier MLLMs exhibit limited zero-shot performance on canine-centric tasks: although state-of-the-art closed-source models outperform open-source counterparts, they still struggle with compositional reasoning over subtle posture and interaction cues spread over long horizons. We observe that generic chain-of-thought prompting provides only modest performance for such long-horizon reasoning. Beyond a novel dataset for canine activity analysis, K9-Bench provides a general-purpose dataset construction pipeline that can be adapted to other low-data domains for quantitative analysis. Our project website is available at: https://ogmenrobotics.github.io/K9Bench.

Problem

• multimodal LLMs
• canine understanding
• video reasoning
• animal-centric AI
• long-horizon reasoning

Innovation

• multimodal LLMs
• canine-centric benchmark
• automated dataset generation
• temporal reasoning
• bias mitigation