DiaLLM: An Investigation into the Robustness-Generation Gap in English Dialect Adaptation

2026-07-08

AI Summary

While large language models demonstrate robust comprehension of English dialects, their generation remains largely confined to Standard American English. This work systematically investigates the adaptation mechanisms for generating Australian, Indian, and Northern British English by applying continued pretraining on international English corpora across three major open-source model families, combined with supervised fine-tuning, explicit and implicit post-training, and three alignment strategies. The study reveals, for the first time, a decoupling between dialect comprehension robustness and generation capability, showing that current evaluation metrics inadequately capture the true efficacy of alignment approaches and highlighting a significant gap between reward optimization and human preferences. Experiments indicate that explicit dialect-directed adaptation yields highly recognizable and preferred outputs, yet excessive dialect reward optimization diminishes preference scores. No single alignment method consistently dominates, underscoring the need for richer reward designs and expanded dialectal resources.

Abstract

Large language models increasingly \emph{understand} dialectal English, yet still \emph{produce} only standard, US-leaning English, leaving dialectal generation, the harder half of the problem, largely unaddressed. We introduce \textbf{DiaLLM}, which continually pretrains three open-weight language model families on the International Corpus of English and applies implicit and explicit post-training paradigms, each combined with three model alignment strategies, giving the first controlled comparison of these components across Australian, Indian, and Northern British English. Our results reveal that dialectal robustness and generation are \emph{dissociated}: benchmarks are shaped by continual pretraining and SFT, while alignment visibly reshapes generation in ways benchmarks do not capture. Explicit variety-targeted adaptation produces output reliably recognised as dialectal and preferred over broad alignment, yet the method that most aggressively optimises the dialectal reward is not preferred by human evaluators. Independent linguistic analysis corroborates this reward-quality gap, most clearly on two of the three families. No single alignment method dominates, and closing the gap will require richer reward designs and continued investment in dialectal resources. We release all code, checkpoints, and preference datasets.

Problem

• dialectal generation
• robustness-generation gap
• English dialect adaptation
• language model alignment
• dialectal reward

Innovation

• dialect adaptation
• robustness-generation gap
• continual pretraining
• model alignment
• preference evaluation