My LLM might Mimic AAE -- But When Should it?

2025-02-06

AI Summary

This study investigates the authenticity and social acceptability of African American English (AAE) generated by large language models (LLMs), addressing longstanding concerns about linguistic bias and representational harm. Method: Employing a Black-led, mixed-methods approach—including large-scale surveys, expert human annotation, context-aware prompt engineering, and multidimensional authenticity evaluation—we systematically examine Black users’ perceptions of and contextual preferences for LLM-generated AAE. Contribution/Results: Findings reveal that, under informally oriented prompts with culturally resonant examples, LLM outputs are rated by Black participants as equally authentic to real-world speech transcripts. Based on this, we propose a “contextualized default strategy”: defaulting to Mainstream American English in formal contexts while enabling AAE generation in informal ones. The work advances principles of linguistic equity and technical empowerment, establishing Black subjectivity as the normative benchmark for AAE modeling and deployment in LLM applications.

Technology Category

• Natural Language Processing: (Large) Language Models
• Machine Learning: Large Multimodal Models (LMMs)
• Humans and AI: AI for Accessibility

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Large language models for search
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems

Abstract

We examine the representation of African American English (AAE) in large language models (LLMs), exploring (a) the perceptions Black Americans have of how effective these technologies are at producing authentic AAE, and (b) in what contexts Black Americans find this desirable. Through both a survey of Black Americans ($n=$ 104) and annotation of LLM-produced AAE by Black Americans ($n=$ 228), we find that Black Americans favor choice and autonomy in determining when AAE is appropriate in LLM output. They tend to prefer that LLMs default to communicating in Mainstream U.S. English in formal settings, with greater interest in AAE production in less formal settings. When LLMs were appropriately prompted and provided in context examples, our participants found their outputs to have a level of AAE authenticity on par with transcripts of Black American speech. Select code and data for our project can be found here: https://github.com/smelliecat/AAEMime.git

Problem

• Examine African American English in LLMs
• Assess Black Americans' perceptions of AAE authenticity
• Determine contexts for AAE use in LLMs

Innovation

• Survey of Black Americans
• Annotation of LLM outputs
• Contextual prompting for AAE