Towards an AI-Driven Video-Based American Sign Language Dictionary: Exploring Design and Usage Experience with Learners

2025-04-08

AI Summary

ASL learners face significant difficulty retrieving unfamiliar signs via text-based search. Method: This paper introduces the first fully automated, video-driven ASL dictionary system: users capture a sign video, and the system performs real-time recognition to return the corresponding lexical entry. We pioneer the integration of state-of-the-art isolated sign language recognition (SLR) with human–computer interaction (HCI) principles, developing an end-to-end deep learning pipeline—including spatiotemporal video feature extraction and cross-modal similarity matching—while iteratively refining the interface to mitigate deployment challenges: AI latency, output uncertainty, re-recording burden, and privacy risks. Contribution/Results: A user study with 12 ASL beginners validates system usability and yields seven empirically grounded design guidelines for human-AI collaboration in sign language tools. This work bridges a critical gap in deployable, video-query ASL dictionaries and establishes both a theoretical framework and practical paradigm for real-world human-centered sign language AI systems.

Technology Category

• Humans and AI: AI for Accessibility
• Computer Vision: Language and Vision
• Natural Language Processing: Conversational AI/Dialog Systems

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Searching for unfamiliar American Sign Language (ASL) signs is challenging for learners because, unlike spoken languages, they cannot type a text-based query to look up an unfamiliar sign. Advances in isolated sign recognition have enabled the creation of video-based dictionaries, allowing users to submit a video and receive a list of the closest matching signs. Previous HCI research using Wizard-of-Oz prototypes has explored interface designs for ASL dictionaries. Building on these studies, we incorporate their design recommendations and leverage state-of-the-art sign-recognition technology to develop an automated video-based dictionary. We also present findings from an observational study with twelve novice ASL learners who used this dictionary during video-comprehension and question-answering tasks. Our results address human-AI interaction challenges not covered in previous WoZ research, including recording and resubmitting signs, unpredictable outputs, system latency, and privacy concerns. These insights offer guidance for designing and deploying video-based ASL dictionary systems.

Problem

• Developing an AI-driven video-based ASL dictionary for learners
• Addressing challenges in searching unfamiliar ASL signs without text queries
• Exploring human-AI interaction issues in video-based sign recognition systems

Innovation

• Automated video-based ASL dictionary system
• State-of-the-art sign-recognition technology
• Addresses human-AI interaction challenges