Evaluation of Pose Estimation Systems for Sign Language Translation

2026-04-27

AI Summary

This study addresses the often-overlooked impact of pose estimation on downstream performance in sign language translation (SLT). Within a unified SLT framework, the authors systematically evaluate eight pose estimation methods—including MediaPipe Holistic, OpenPose, and MMPose—on the RWTH-PHOENIX-Weather 2014 and Signsuisse datasets. A multidimensional analysis is conducted across temporal stability, keypoint completeness, and robustness to occlusion. Results demonstrate that SDPose and Sapiens achieve the best translation performance (BLEU ≈ 11.5), significantly outperforming the MediaPipe baseline (BLEU ≈ 10). Notably, Sapiens correctly translates all 15 occluded samples, whereas OpenPifPaf succeeds in only 1 out of 15 and yields the lowest scores. This work provides the first comprehensive investigation into the relationship between pose estimation quality and SLT performance and releases a complete open-source toolchain to support future research.

Technology Category

• Computer Vision: Biometrics, Face, Gesture & Pose
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Intelligent Robots: State Estimation

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Systems and Infrastructure for Web, Mobile and WoT: Web performance, measurement, and characterization
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

Many sign language translation (SLT) systems operate on pose sequences instead of raw video to reduce input dimensionality, improve portability, and partially anonymize signers. The choice of pose estimator is often treated as an implementation detail, with systems defaulting to widely available tools such as MediaPipe Holistic or OpenPose. We present a systematic comparison of pose estimators for pose-based SLT, covering widely used baselines (MediaPipe Holistic, OpenPose) and newer whole-body/high-capacity models (MMPose WholeBody, OpenPifPaf, AlphaPose, SDPose, Sapiens, SMPLest-X). We quantify downstream impact by training a controlled SLT pipeline on RWTH-PHOENIX-Weather 2014 where only the pose representation varies, evaluating with BLEU and BLEURT. To contextualize translation outcomes, we analyze temporal stability, missing hand keypoints, and robustness to occlusion using higher-resolution videos from the Signsuisse dataset. SDPose and Sapiens achieve the best translation performance (BLEU ~11.5), outperforming the common MediaPipe baseline (BLEU ~10). In occlusion cases, Sapiens is correct in all tested instances (15/15), while OpenPifPaf fails in nearly all (1/15) and also yields the weakest translation scores. Estimators that frequently leave out hand keypoints are associated with lower BLEU/BLEURT. We release code that can be used not only to reproduce our experiments, but also considerably lowers the barrier for other researchers to use alternative pose estimators.

Problem

• pose estimation
• sign language translation
• hand keypoints
• occlusion robustness
• temporal stability

Innovation

• pose estimation
• sign language translation
• occlusion robustness
• keypoint stability
• systematic evaluation