Are they lovers or friends? Evaluating LLMs' Social Reasoning in English and Korean Dialogues

2025-10-21

AI Summary

This study investigates large language models’ (LLMs) capacity for cross-lingual social reasoning—specifically, inferring interpersonal relationships (e.g., romantic partners, friends, sisters) from bilingual dialogues. To this end, we introduce SCRIPTS, the first bilingual benchmark for social relation inference, comprising 1,000 human-annotated English–Korean movie dialogues with fine-grained, three-tier probabilistic labels (highly likely / unlikely / impossible). Experimental results show that leading closed-source LLMs achieve 75–80% accuracy on English inputs but only 58–69% on Korean—revealing substantial linguistic and cultural biases. Notably, 10–25% of erroneous predictions are confidently assigned “impossible” labels, indicating systematic overconfidence in culturally misaligned inferences. Moreover, chain-of-thought prompting fails to improve performance and exacerbates bias. This work provides the first systematic empirical evidence of structural deficiencies in LLMs’ cross-cultural social reasoning, establishing a novel benchmark and foundational insights for trustworthy AI and computational social cognition.

Technology Category

• Natural Language Processing: (Large) Language Models
• Machine Learning: Large Multimodal Models (LMMs)
• Cognitive Modeling & Cognitive Systems: Social Cognition And Interaction

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

As large language models (LLMs) are increasingly used in human-AI interactions, their social reasoning capabilities in interpersonal contexts are critical. We introduce SCRIPTS, a 1k-dialogue dataset in English and Korean, sourced from movie scripts. The task involves evaluating models' social reasoning capability to infer the interpersonal relationships (e.g., friends, sisters, lovers) between speakers in each dialogue. Each dialogue is annotated with probabilistic relational labels (Highly Likely, Less Likely, Unlikely) by native (or equivalent) Korean and English speakers from Korea and the U.S. Evaluating nine models on our task, current proprietary LLMs achieve around 75-80% on the English dataset, whereas their performance on Korean drops to 58-69%. More strikingly, models select Unlikely relationships in 10-25% of their responses. Furthermore, we find that thinking models and chain-of-thought prompting, effective for general reasoning, provide minimal benefits for social reasoning and occasionally amplify social biases. Our findings reveal significant limitations in current LLMs' social reasoning capabilities, highlighting the need for efforts to develop socially-aware language models.

Problem

• Evaluating LLMs' social reasoning in multilingual dialogues
• Assessing interpersonal relationship inference capabilities
• Identifying performance gaps between English and Korean

Innovation

• Dataset SCRIPTS with bilingual movie dialogues
• Probabilistic labels for interpersonal relationship inference
• Evaluating models' social reasoning across cultures