MouseGPT: A Large-scale Vision-Language Model for Mouse Behavior Analysis

2025-03-13

AI Summary

Traditional mouse behavioral analysis relies on manual annotation, suffering from poor interpretability and limited generalizability—hindering systematic characterization of the neurobehavioral spectrum. To address this, we introduce the first large-scale vision-language model (VLM) specifically designed for mouse behavior. Our approach integrates self-supervised pose representation learning with open-vocabulary behavioral prompting, implemented within a custom multimodal architecture trained on 42 million frames of multi-pathological behavioral videos. The model enables fine-grained behavior recognition, unsupervised behavioral clustering, and discovery of novel behaviors—all without manual labels—while supporting context-aware semantic interpretation of behavior. It outperforms existing methods across behavioral classification accuracy, cross-experimental robustness, and descriptive richness. This work establishes a high-throughput, interpretable phenotyping paradigm for neuropsychiatric disorders.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Analyzing animal behavior is crucial in advancing neuroscience, yet quantifying and deciphering its intricate dynamics remains a significant challenge. Traditional machine vision approaches, despite their ability to detect spontaneous behaviors, fall short due to limited interpretability and reliance on manual labeling, which restricts the exploration of the full behavioral spectrum. Here, we introduce MouseGPT, a Vision-Language Model (VLM) that integrates visual cues with natural language to revolutionize mouse behavior analysis. Built upon our first-of-its-kind dataset - incorporating pose dynamics and open-vocabulary behavioral annotations across over 42 million frames of diverse psychiatric conditions - MouseGPT provides a novel, context-rich method for comprehensive behavior interpretation. Our holistic analysis framework enables detailed behavior profiling, clustering, and novel behavior discovery, offering deep insights without the need for labor - intensive manual annotation. Evaluations reveal that MouseGPT surpasses existing models in precision, adaptability, and descriptive richness, positioning it as a transformative tool for ethology and for unraveling complex behavioral dynamics in animal models.

Problem

• Quantifying and deciphering intricate mouse behavior dynamics.
• Overcoming limitations of traditional machine vision in behavior analysis.
• Providing a context-rich method for comprehensive behavior interpretation.

Innovation

• Integrates visual cues with natural language
• Utilizes a large-scale dataset with diverse conditions
• Enables detailed behavior profiling without manual annotation