Abjad AI at NADI 2025: CATT-Whisper: Multimodal Diacritic Restoration Using Text and Speech Representations

2025-10-28

AI Summary

This paper addresses the diacritization (Tashkeel) restoration task for Arabic dialectal sentences. We propose a multimodal approach integrating textual and phonetic modalities: a CATT-based text encoder is jointly trained with a Whisper speech encoder, employing both early feature fusion and cross-modal attention mechanisms; additionally, a random speech input masking strategy is introduced to improve robustness and generalization. Evaluated on standard dialectal benchmarks, our model achieves 0.25 WER / 0.90 CER on the development set and 0.55 WER / 0.13 CER on the test set—substantially outperforming unimodal baselines. To the best of our knowledge, this is the first work to incorporate large-scale pretrained speech encoders into Arabic dialectal Tashkeel restoration, demonstrating that multimodal joint modeling significantly enhances phoneme–grapheme alignment in low-resource dialectal settings.

Technology Category

• Natural Language Processing: Language Grounding & Multi-modal NLP
• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Multi-modal Vision

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs

Abstract

In this work, we tackle the Diacritic Restoration (DR) task for Arabic dialectal sentences using a multimodal approach that combines both textual and speech information. We propose a model that represents the text modality using an encoder extracted from our own pre-trained model named CATT. The speech component is handled by the encoder module of the OpenAI Whisper base model. Our solution is designed following two integration strategies. The former consists of fusing the speech tokens with the input at an early stage, where the 1500 frames of the audio segment are averaged over 10 consecutive frames, resulting in 150 speech tokens. To ensure embedding compatibility, these averaged tokens are processed through a linear projection layer prior to merging them with the text tokens. Contextual encoding is guaranteed by the CATT encoder module. The latter strategy relies on cross-attention, where text and speech embeddings are fused. The cross-attention output is then fed to the CATT classification head for token-level diacritic prediction. To further improve model robustness, we randomly deactivate the speech input during training, allowing the model to perform well with or without speech. Our experiments show that the proposed approach achieves a word error rate (WER) of 0.25 and a character error rate (CER) of 0.9 on the development set. On the test set, our model achieved WER and CER scores of 0.55 and 0.13, respectively.

Problem

• Restoring diacritics in Arabic dialectal sentences
• Combining text and speech representations multimodally
• Improving robustness with optional speech input during training

Innovation

• Multimodal fusion of text and speech for diacritic restoration
• Cross-attention mechanism integrates text and speech embeddings
• Speech input randomly deactivated during training for robustness