RatBodyFormer: Rat Body Surface from Keypoints

2024-12-12

AI Summary

Existing 3D pose estimation methods rely on sparse keypoints (e.g., facial landmarks, limb joints) and thus fail to capture fine-grained rat behaviors—such as curling or stretching—that manifest primarily through dense, non-rigid surface deformations, owing to the lack of stable visual features on the rat’s fur-covered body surface. To address this, we propose RatBodyFormer: the first end-to-end framework for reconstructing dense, millimeter-accurate body surface point clouds from sparse keypoints specifically designed for rats. Our method leverages a custom-built multi-view RatDome acquisition system and a large-scale paired dataset of surface points and anatomical keypoints. It employs a keypoint-guided Transformer architecture with position-agnostic point prediction and a self-supervised masked surface reconstruction objective. Experiments demonstrate high-fidelity, generalizable, and deployable 3D surface reconstruction in real-world settings—enabling, for the first time, robust, fine-grained behavioral analysis at the body-surface level.

Technology Category

• Computer Vision: 3D Computer Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding

Application Category

• Semantics and Knowledge: Scalable techniques for the creation, curation, publication, maintenance, and consumption of large, Web-based, structured, reusable, knowledge graphs and ontologies
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs

Abstract

Analyzing rat behavior lies at the heart of many scientific studies. Past methods for automated rodent modeling have focused on 3D pose estimation from keypoints, e.g., face and appendages. The pose, however, does not capture the rich body surface movement encoding the subtle rat behaviors like curling and stretching. The body surface lacks features that can be visually defined, evading these established keypoint-based methods. In this paper, we introduce the first method for reconstructing the rat body surface as a dense set of points by learning to predict it from the sparse keypoints that can be detected with past methods. Our method consists of two key contributions. The first is RatDome, a novel multi-camera system for rat behavior capture, and a large-scale dataset captured with it that consists of pairs of 3D keypoints and 3D body surface points. The second is RatBodyFormer, a novel network to transform detected keypoints to 3D body surface points. RatBodyFormer is agnostic to the exact locations of the 3D body surface points in the training data and is trained with masked-learning. We experimentally validate our framework with a number of real-world experiments. Our results collectively serve as a novel foundation for automated rat behavior analysis.

Problem

• Reconstructing rat body surface from sparse keypoints
• Capturing subtle behaviors like curling and stretching
• Overcoming limitations of keypoint-based methods

Innovation

• Predicts dense body surface from sparse keypoints
• Introduces RatDome multi-camera capture system
• Uses masked-learning for agnostic surface reconstruction