InterPet4D: A Multimodal 4D Human-Pet Interaction Dataset for Pet Motion Generation

2026-07-11

AI Summary

This study addresses the lack of high-quality, large-scale datasets capturing human–pet interactions by introducing InterPet4D, the first 4D multimodal dataset of natural human–dog interactions. It comprises synchronized multi-view videos, audio, 2D/3D keypoints, meshes, and segmentation masks from 13 dogs and 23 humans performing obedience tasks. Leveraging this dataset, the authors propose InterPetMoGen, a dedicated framework for generating human–pet interactive motions. Experimental results demonstrate that InterPetMoGen achieves an FID score of 11.21 on InterPet4D, significantly outperforming baseline methods such as Seq2Seq and DiT, thereby validating both the dataset’s utility and the framework’s state-of-the-art performance in modeling and generating realistic human–pet interactions.

Abstract

Human-pet interaction estimation and generation remain underexplored due to the absence of a high-quality large-scale dataset. We present InterPet4D, the first multimodal dataset capturing natural interactions between humans and dogs. Using a synchronized multi-view capture system, we record human-dog obedience tasks and provide annotations for both humans and dogs, including multi-view and egocentric videos, segmentations, 2D and 3D keypoints, meshes, and audio tracks. InterPet4D consists of 6.8 million frames collected from 13 dogs of 11 breeds interacting with 23 human participants. We further introduce the InterPetMoGen framework for human-pet interaction motion generation. Our proposed model achieves an FID score of 11.21 and substantially outperforms the Seq2Seq and DiT baselines, demonstrating the effectiveness of InterPet4D for modeling realistic human-pet interactions.

Problem

• human-pet interaction
• motion generation
• multimodal dataset
• 4D interaction
• pet behavior

Innovation

• multimodal dataset
• 4D human-pet interaction
• motion generation
• synchronized multi-view capture
• InterPetMoGen