Meaningful Pose-Based Sign Language Evaluation

2025-10-08

AI Summary

This study addresses the lack of reliable automated evaluation methods for sign language AI systems by proposing the first multi-dimensional sign language expression assessment framework grounded in human skeletal pose. Methodologically, it integrates three complementary metrics—keypoint geometric distance, pose embedding similarity, and cross-modal back-translation quality—to establish an interpretable and reproducible evaluation pipeline. Through automated meta-evaluation and cross-lingual human correlation studies, the framework systematically characterizes the applicability boundaries of each metric across sign-to-text retrieval and text-to-pose generation tasks. Key contributions include: (1) uncovering systematic performance trade-offs among evaluation metrics; (2) releasing PoseEval—an open-source, modular pose evaluation toolkit; and (3) advancing standardization in sign language assessment, thereby significantly improving development and iterative refinement efficiency for sign language translation and generation systems.

Technology Category

• Machine Learning: Evaluation and Analysis
• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models
• Computer Vision: Biometrics, Face, Gesture & Pose

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success
• Systems and Infrastructure for Web, Mobile and WoT: Web performance, measurement, and characterization

Abstract

We present a comprehensive study on meaningfully evaluating sign language utterances in the form of human skeletal poses. The study covers keypoint distance-based, embedding-based, and back-translation-based metrics. We show tradeoffs between different metrics in different scenarios through automatic meta-evaluation of sign-level retrieval and a human correlation study of text-to-pose translation across different sign languages. Our findings and the open-source pose-evaluation toolkit provide a practical and reproducible way of developing and evaluating sign language translation or generation systems.

Problem

• Evaluating sign language utterances using skeletal poses
• Comparing keypoint distance, embedding, and back-translation metrics
• Developing reproducible evaluation methods for sign language systems

Innovation

• Evaluates sign language using skeletal pose metrics
• Compares keypoint distance and embedding-based approaches
• Provides open-source toolkit for reproducible evaluation