Phonological Perception of Sign Language Models

2026-06-26

AI Summary

This study investigates whether sign language recognition models genuinely comprehend the phonological structure of American Sign Language or merely rely on low-level statistical regularities. Through minimal pair testing and representational alignment with human behavioral data, the authors systematically evaluate the sensitivity of both pose-based and pixel-based deep learning models to critical phonological parameters—such as handshape and location. The work reveals, for the first time, a marked divergence in phonological sensitivity between the two model types: pose-based models exhibit greater sensitivity to handshape variations, with internal representations significantly correlated with human perception (r ≈ 0.49), whereas pixel-based models are more adept at encoding location information. These findings offer a novel cognitively aligned perspective for computational modeling of sign language.

Abstract

Sign languages are compositional systems where meaning arises by combining sublexical phonological parameters, such as handshape, location, and movement. While deep learning models for Sign Language Recognition (SLR) have achieved increased performance on translation benchmarks, it remains unclear whether these models distinguish abstract phonological features or merely rely on low-level statistical correlations. This work evaluates the phonological perception of SLR models trained on American Sign Language (ASL) by probing phonological sensitivity using minimal pairs and evaluating representational alignment with human behavioral data. Our results reveal that SLR models exhibit emergent phonological sensitivity, but with clear architectural trade-offs: pose-based models are sensitive to handshape contrasts, while pixel-based models better capture location changes. Furthermore, pose-based models learn latent representations that correlate with human perceptual similarity judgments (r~0.49). These findings suggest that while SLR models exhibit emergent phonology, current training paradigms are insufficient to scale them beyond their architectural inductive biases.

Problem

• Sign Language Recognition
• Phonological Perception
• Minimal Pairs
• Representational Alignment
• American Sign Language

Innovation

• phonological perception
• sign language recognition
• minimal pairs
• representational alignment
• inductive bias