CANDLE: Character-level Arabic Noise Deduplication using Lightweight Encoder

2026-06-23

AI Summary

This work addresses the challenge of distinguishing between legitimate orthographic repetition and informal, elongated character noise in Arabic social media text. To this end, it introduces Connectionist Temporal Classification (CTC) to the character-level deduplication task for the first time, formulating it as a sequence alignment problem grounded in character-level encoding and realized through a lightweight end-to-end architecture for noise normalization. Leveraging knowledge distillation, the model achieves a threefold compression in depth with negligible performance degradation, substantially reducing inference overhead. Experimental results demonstrate that the proposed approach attains a Sentence Error Rate as low as 5.37% across three benchmarks and reduces the fertility of downstream tokenizers by up to 12.8%.

Abstract

Handling repeated characters in text can be tricky, since they can represent either the correct spelling of a word or informal character elongation often seen in social media posts. We present CANDLE, a lightweight system for character-level Arabic noise deduplication that addresses this challenge without relying on handcrafted rules, dictionaries, or morphological analyzers. At the heart of CANDLE is a novel application of Connectionist Temporal Classification (CTC) to this task, a formulation not previously explored for character deduplication, which frames normalization as a sequence alignment problem over a character-based encoder. Evaluated on three benchmarks spanning clean newspaper, manually curated ambiguous cases, and real-world social media text, the CTC model achieves a Sentence Error Rate (SER) as low as $5.37\%$ and consistently outperforms a classification-based baseline by a large margin. To reduce inference overhead, we distill the 6-layer CTC model into a 2-layer student, achieving a $3\times$ depth reduction with minimal performance degradation. Beyond deduplication accuracy, normalization yields a practical downstream benefit: a relative reduction in tokenizer fertility of up to $12.8\%$ across a diverse set of Arabic LLM tokenizers, directly lowering inference costs and improving context window utilization. We release all code and models publicly to support reproducibility and advance future research\footnote{https://github.com/abjadai/candle}.

Problem

• Arabic text normalization
• character-level deduplication
• repeated characters
• noise handling
• social media text

Innovation

• Connectionist Temporal Classification
• character-level deduplication
• Arabic text normalization
• model distillation
• tokenizer fertility