Reading between the Lines: Can LLMs Identify Cross-Cultural Communication Gaps?

2025-02-08

AI Summary

This work addresses cross-cultural comprehension barriers in book reviews under globalization, systematically investigating how culture-specific expressions impair reader understandability. Drawing on 57 Goodreads reviews, we integrate expert annotation, cross-cultural user experiments, and zero-shot GPT-4o evaluation—revealing that 83% of reviews contain culturally opaque elements and introducing the first publicly available benchmark for Cultural Understanding Bias (CUB). Our analysis exposes significant instability in large language models’ ability to identify context-appropriate cultural grounding, with suboptimal accuracy. Key contributions are: (1) empirical quantification of the prevalence of cultural comprehension barriers in book reviews; (2) a reproducible, multi-source validation methodology combining human judgment, behavioral data, and LLM-based assessment; and (3) open-sourcing of the CUB benchmark dataset and evaluation protocol, establishing foundational infrastructure for culture-aware NLP research.

Technology Category

• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models
• Computer Vision: Bias, Fairness & Privacy
• Machine Learning: Evaluation and Analysis

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

In a rapidly globalizing and digital world, content such as book and product reviews created by people from diverse cultures are read and consumed by others from different corners of the world. In this paper, we investigate the extent and patterns of gaps in understandability of book reviews due to the presence of culturally-specific items and elements that might be alien to users from another culture. Our user-study on 57 book reviews from Goodreads reveal that 83% of the reviews had at least one culture-specific difficult-to-understand element. We also evaluate the efficacy of GPT-4o in identifying such items, given the cultural background of the reader; the results are mixed, implying a significant scope for improvement. Our datasets are available here: https://github.com/sougata-ub/reading_between_lines

Problem

• Identifies cross-cultural communication gaps in reviews.
• Evaluates GPT-4o's ability to detect cultural-specific elements.
• Assesses understandability of culturally diverse book reviews.

Innovation

• LLMs identify cultural gaps
• GPT-4o evaluates cultural elements
• User-study analyzes book reviews