Tracking Mouse from Incomplete Body-Part Observations and Deep-Learned Deformable-Mouse Model Motion-Track Constraint for Behavior Analysis

2025-01-19

AI Summary

To address incomplete 3D vertebral tracking in multi-view mouse videos caused by occlusion, this paper proposes a geometry- and learning-integrated optimization framework. First, initial vertebral observations are obtained via multi-camera extrinsic calibration and 3D triangulation. Then, a deformable, anatomy-consistent 3D mouse model is introduced, jointly incorporating joint motion priors modeled by a graph convolutional network and physics-constrained nonlinear trajectory optimization to achieve globally smooth, occlusion-robust whole-body vertebral tracking. This work is the first to embed a deep learning–driven joint prior into a deformable anatomical model-guided bundle adjustment framework. In neuroethological experiments, it improves vertebral tracking completeness by over 42%, significantly enhancing segmentation accuracy for behaviors such as grasping and exploration, and increasing the reliability of behavioral phenotyping quantification.

Technology Category

• Computer Vision: Motion & Tracking
• Search and Optimization: Learning to Search
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Tracking mouse body parts in video is often incomplete due to occlusions such that - e.g. - subsequent action and behavior analysis is impeded. In this conceptual work, videos from several perspectives are integrated via global exterior camera orientation; body part positions are estimated by 3D triangulation and bundle adjustment. Consistency of overall 3D track reconstruction is achieved by introduction of a 3D mouse model, deep-learned body part movements, and global motion-track smoothness constraint. The resulting 3D body and body part track estimates are substantially more complete than the original single-frame-based body part detection, therefore, allowing improved animal behavior analysis.

Problem

• Mouse behavior analysis
• Occlusion handling
• Tracking completeness

Innovation

• 3D tracking
• multi-view video data
• deep learning