Semantic Hardness Is Not Visual Hardness: Sign-Aware Hard Negative Mining for Sign Language Retrieval

2026-07-10

AI Summary

This work addresses the challenge of fine-grained sign language retrieval, where visually similar but semantically distinct samples are difficult to discriminate. Existing approaches rely on textual semantics to construct hard negatives, which often fail to align with visual ambiguities inherent in sign language. To overcome this limitation, the paper proposes Sign-language-Aware Hard Negative mining (SAN), which reveals for the first time the inconsistency between semantic and visual difficulty. SAN directly constructs hard negatives in the sign embedding space based on visual confusability, bypassing the constraints of text-driven methods. Integrated with contrastive learning, the proposed approach significantly improves fine-grained retrieval performance on PHOENIX-2014T while maintaining accuracy in coarse-grained retrieval.

Abstract

Sign Language Retrieval (SLRet) enables efficient access to sign language content but remains fragile in fine-grained scenarios where visually similar signs must be distinguished. We show that this limitation does not stem from model capacity, but from ineffective hard negative supervision. Specifically, we formulate fine-grained retrieval failures as a negative distribution mismatch: semantically distinct yet visually confusable signs are rarely treated as hard negatives, while existing text-based mining strategies fail to capture such visual ambiguity. To address this issue, we propose Sign-Aware Hard Negative Mining (SAN), which constructs hard negatives based on visual confusability in the sign embedding space rather than linguistic similarity. Experiments on PHOENIX-2014T demonstrate that SAN substantially improves fine-grained retrieval performance while preserving coarse-grained accuracy, highlighting the importance of aligning negative supervision with visual ambiguity in sign language retrieval.

Problem

• Sign Language Retrieval
• Hard Negative Mining
• Visual Confusability
• Fine-grained Retrieval
• Negative Distribution Mismatch

Innovation

• hard negative mining
• sign language retrieval
• visual confusability
• embedding space
• fine-grained retrieval