The In-Car Sign Language Corpus (ICSL): A Multi-Modal Resource for Constrained-Space Sign Language Recognition

2026-07-13

AI Summary

This work addresses the scarcity of research on sign language recognition in constrained, occlusion-prone, and non-frontal-view settings—such as shared mobility environments—by introducing ICSL, the first in-car multimodal corpus for Brazilian Sign Language (Libras). ICSL integrates synchronized high-fidelity motion capture data from controlled laboratory conditions with real-world 2D/3D sensor recordings captured inside vehicles, comprising over 1.5 million annotated frames labeled for both lexical and non-lexical linguistic elements. This corpus enables the training, evaluation, and domain adaptation of sign language recognition models under spatially restricted conditions. By providing paired idealized and real-world data, ICSL facilitates comparative analysis between synthetic avatars and human signer videos, establishing a high-quality foundation for developing accessible in-vehicle human–computer interaction technologies.

Abstract

This paper addresses the challenges of using sign language within shared mobility services, such as taxis, carpools, or ride-sharing platforms. The use of sign language recognition (SLR) in real-world, confined environments, specifically vehicle interiors remains largely unexplored. To motivate research in this area, we present the In-Car Sign Language (ICSL) dataset for Brazilian Sign Language (Libras), with the long-term goal of improving public transport accessibility for the Deaf and Hard-of-Hearing community. The dataset consists of: (1) high-precision laboratory motion capture (MoCap) data to establish an idealized linguistic baseline and (2) real-world multi-modal in-car recordings captured using a 2D camera and 3D Time-of-Flight sensors. The dataset provides a basis for comparative analyses between synthesized signing avatar animations and recorded real signing interpreter videos, which enable future research into robust "in-the-wild" SLR models and domain adaptation. We describe in detail the use cases, the setup, the data collection protocol, and the metadata structure of the corpus. In total, we recorded a multimodal dataset exceeding 1.5 million frames, comprising the synchronized multimodal streams described above featuring Libras users across various in-car scenarios. The corpus is provided with gloss annotation of lexical signs and non-lexical sign language elements specially designed to support the training and evaluation of deep neural networks for constrained space recognition. In-vehicle signing offers a technically significant example of a constrained, occluded, and non-frontal environment. While recognizing the diverse communication strategies already employed by the Deaf community, identifying automotive-specific limitations provides a useful stepping stone for research into enhancing in-car accessibility and passenger quality of life.

Problem

• sign language recognition
• constrained-space
• in-car environment
• accessibility
• shared mobility

Innovation

• sign language recognition
• in-car environment
• multimodal dataset
• constrained-space
• domain adaptation