Fish2Mesh Transformer: 3D Human Mesh Recovery from Egocentric Vision

2025-03-08

AI Summary

This work addresses the challenging problem of 3D human mesh recovery from egocentric fisheye camera views, tackling three key difficulties: severe self-occlusion, strong fisheye distortion, and scarcity of real-world annotated data. To this end, we propose a fisheye-aware positional embedding module—first integrating fisheye geometric priors into a Transformer architecture—and design a multi-task joint regression framework that simultaneously optimizes SMPL pose/shape parameters and camera translation. Furthermore, we construct a weakly supervised synthetic dataset driven by 4D-Human, incorporating fisheye-corrected positional encoding and auxiliary 2D/3D joint losses. On the egocentric HMR benchmark, our method significantly outperforms state-of-the-art approaches, achieving notable improvements in both 3D mesh accuracy and surface geometric consistency.

Technology Category

• Computer Vision: 3D Computer Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Egocentric human body estimation allows for the inference of user body pose and shape from a wearable camera's first-person perspective. Although research has used pose estimation techniques to overcome self-occlusions and image distortions caused by head-mounted fisheye images, similar advances in 3D human mesh recovery (HMR) techniques have been limited. We introduce Fish2Mesh, a fisheye-aware transformer-based model designed for 3D egocentric human mesh recovery. We propose an egocentric position embedding block to generate an ego-specific position table for the Swin Transformer to reduce fisheye image distortion. Our model utilizes multi-task heads for SMPL parametric regression and camera translations, estimating 3D and 2D joints as auxiliary loss to support model training. To address the scarcity of egocentric camera data, we create a training dataset by employing the pre-trained 4D-Human model and third-person cameras for weak supervision. Our experiments demonstrate that Fish2Mesh outperforms previous state-of-the-art 3D HMR models.

Problem

• Recover 3D human mesh from fisheye egocentric images.
• Address self-occlusions and distortions in fisheye images.
• Overcome scarcity of egocentric data for training.

Innovation

• Fisheye-aware transformer for 3D mesh recovery
• Egocentric position embedding reduces fisheye distortion
• Multi-task heads for SMPL regression and camera translations