DialectGen: Benchmarking and Improving Dialect Robustness in Multimodal Generation

2025-10-16

AI Summary

This study identifies a significant performance degradation of contemporary multimodal generative models (text-to-image/video) when conditioned on dialectal text prompts. To address this, we introduce the first large-scale benchmark covering six English dialects and systematically evaluate 17 state-of-the-art models, revealing widespread deficiencies in dialectal robustness. We propose a general encoder-enhancement framework integrating dialect speaker–informed prompt collection, hybrid human-automated evaluation, prompt rewriting, and lightweight fine-tuning—designed to improve dialect adaptation without compromising standard American English performance. Experiments demonstrate that our method boosts Stable Diffusion 1.5’s generation quality by an average of 34.4% across five dialects, bringing it close to standard English performance. This validates the effectiveness and generalizability of encoder-level dialectal robustness modeling.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Generation
• Computer Vision: Diffusion Models for Vision

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Web Mining and Content Analysis: Large pretrained models with web data
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Contact languages like English exhibit rich regional variations in the form of dialects, which are often used by dialect speakers interacting with generative models. However, can multimodal generative models effectively produce content given dialectal textual input? In this work, we study this question by constructing a new large-scale benchmark spanning six common English dialects. We work with dialect speakers to collect and verify over 4200 unique prompts and evaluate on 17 image and video generative models. Our automatic and human evaluation results show that current state-of-the-art multimodal generative models exhibit 32.26% to 48.17% performance degradation when a single dialect word is used in the prompt. Common mitigation methods such as fine-tuning and prompt rewriting can only improve dialect performance by small margins (<7%), while potentially incurring significant performance degradation in Standard American English (SAE). To this end, we design a general encoder-based mitigation strategy for multimodal generative models. Our method teaches the model to recognize new dialect features while preserving SAE performance. Experiments on models such as Stable Diffusion 1.5 show that our method is able to simultaneously raise performance on five dialects to be on par with SAE (+34.4%), while incurring near zero cost to SAE performance.

Problem

• Evaluating multimodal generative models' dialect robustness
• Assessing performance degradation with dialectal text inputs
• Developing mitigation strategy for dialect-SAE performance balance

Innovation

• Developed an encoder-based dialect robustness strategy
• Enabled multimodal models to recognize dialect features
• Maintained Standard American English performance while improving dialects