TSR-Ego: Temporally Guided Stereo Refinement Framework for Egocentric 3D Human Pose Estimation

2026-07-10

AI Summary

This work addresses the challenging problem of egocentric 3D human pose estimation from head-mounted fisheye stereo cameras, where severe self-occlusion, limb truncation, and lens distortion degrade performance. To tackle these issues, we propose TSR-Ego, a novel framework that uniquely integrates causal depth-wise temporal convolutions with fisheye distortion-aware stereo feature sampling. Within a single-stage causal stereo decoder, our approach jointly leverages temporal self-attention, joint-wise self-attention, and deformable stereo cross-attention to enable online, future-frame-free 3D pose refinement. By coherently modeling both temporal motion context and stereo geometric cues, TSR-Ego significantly enhances robustness under weak observation and heavy occlusion. The method achieves state-of-the-art performance on the UnrealEgo2 and UnrealEgo-RW benchmarks, demonstrating particularly pronounced advantages on real-world sequences.

Abstract

Egocentric 3D human pose estimation from head-mounted stereo cameras is challenging due to fisheye distortion, severe self-occlusion, and frequent truncation of body joints outside the camera field of view. Recent stereo egocentric methods have improved performance through heatmap lifting, stereo correspondence, and transformer-based refinement, but they often rely heavily on frame-local evidence or use temporal information only as auxiliary pose-level context. This limits robustness when current-frame stereo cues are weak, occluded, or ambiguous. We propose TSR-Ego, a temporally guided stereo framework that couples short-term motion evidence with projection-guided feature sampling. The model first enriches dense stereo feature maps using a causal depthwise-separable temporal convolution, allowing past visual evidence to influence the feature space before deformable cross-attention. A single-stage causal stereo decoder then refines learned 3D joint queries through temporal self-attention, joint self-attention, and fisheye deformable stereo cross-attention, using the evolving pose estimate to generate 2D sampling references. Unlike methods that apply temporal reasoning mainly after pose prediction, TSR-Ego uses motion context to shape both the sampled stereo features and the joint representations while preserving online inference without future frames. Experiments on UnrealEgo2 and UnrealEgo-RW show state-of-the-art performance, with especially strong gains on real-world sequences.

Problem

• egocentric 3D human pose estimation
• stereo cameras
• self-occlusion
• fisheye distortion
• temporal robustness

Innovation

• temporal guidance
• stereo refinement
• egocentric 3D pose estimation
• causal convolution
• deformable cross-attention