Can Dialects Be Steered Like Languages? Sparse Neurons and Distributed Directions in Arabic LLMs

2026-07-04

AI Summary

This work addresses the challenge that large language models for Arabic tend to overgenerate Modern Standard Arabic (MSA) due to the scarcity of dialectal data, thereby struggling to produce accurate dialectal outputs. The study is the first to reveal that dialectal information within these models is jointly encoded through sparse neuron activations and distributed directional representations in the embedding space. Building on this insight, the authors propose a novel inference-time control framework that requires no fine-tuning: by identifying critical neurons and manipulating dialect-associated directions in the vector space, the method dynamically steers the generation process toward the target dialect. This approach significantly improves dialectal generation accuracy and offers an interpretable, efficient solution for controllable text generation in languages with multiple linguistic varieties.

Abstract

A key challenge in Arabic NLP is the scarcity of dialectal data relative to Modern Standard Arabic (MSA), causing LLMs to overproduce MSA and struggle with dialectally accurate generation. From an interpretability perspective, this raises a fundamental question: where and how are dialectal features encoded within model internals, and can these representations be leveraged to improve dialect generation without fine-tuning? This study investigates two complementary inference-time approaches that serve simultaneously as interpretability probes and control mechanisms. First, we conduct a neuron-level analysis, identifying sparse neuron populations that encode dialect-specific features and showing that amplifying or suppressing these neurons can steer model outputs toward target dialects. Second, motivated by the entanglement of dialectal features at the single-neuron level, we apply a vector-steering approach that extracts dialect-specific activation directions and injects them during inference. Together, these methods illuminate the geometry of dialectal knowledge in Arabic LLMs and offer a principled, interpretability-grounded framework for dialect control without requiring dialect-specific fine-tuning.

Problem

• Arabic dialects
• dialect generation
• language models
• data scarcity
• Modern Standard Arabic

Innovation

• sparse neurons
• vector steering
• dialect control
• Arabic LLMs
• interpretability