EMODIS: A Benchmark for Context-Dependent Emoji Disambiguation in Large Language Models

2025-11-10

AI Summary

This work addresses the underexplored problem of large language models’ (LLMs) ability to disambiguate ambiguous emojis in minimal contrastive textual contexts. To this end, the authors introduce EmojiDisambiguation—the first benchmark specifically designed for evaluating context-dependent emoji disambiguation—built upon human-crafted triplets (ambiguous sentence–contrastive context–reasoning question). It adopts a linguistically inspired task framework and employs joint evaluation across diverse open- and closed-source LLMs. The key contribution is a contrastive context-pair design that systematically reveals two pervasive LLM biases: insensitivity to subtle pragmatic cues and strong preference for dominant interpretations. Experiments show even state-of-the-art models frequently fail on fine-grained contextual distinctions, highlighting a substantial gap between LLMs and human semantic reasoning capabilities. The benchmark provides a reproducible, extensible testbed for future research on emoji semantics and contextual inference.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Humans and AI: Emotional Intelligence

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling

Abstract

Large language models (LLMs) are increasingly deployed in real-world communication settings, yet their ability to resolve context-dependent ambiguity remains underexplored. In this work, we present EMODIS, a new benchmark for evaluating LLMs'capacity to interpret ambiguous emoji expressions under minimal but contrastive textual contexts. Each instance in EMODIS comprises an ambiguous sentence containing an emoji, two distinct disambiguating contexts that lead to divergent interpretations, and a specific question that requires contextual reasoning. We evaluate both open-source and API-based LLMs, and find that even the strongest models frequently fail to distinguish meanings when only subtle contextual cues are present. Further analysis reveals systematic biases toward dominant interpretations and limited sensitivity to pragmatic contrast. EMODIS provides a rigorous testbed for assessing contextual disambiguation, and highlights the gap in semantic reasoning between humans and LLMs.

Problem

• Evaluating LLMs' ability to interpret ambiguous emoji expressions in minimal contexts
• Assessing models' capacity to resolve context-dependent ambiguity using contrastive cues
• Benchmarking semantic reasoning gaps between humans and LLMs in emoji disambiguation

Innovation

• Benchmark evaluates emoji disambiguation in contexts
• Tests models with minimal contrastive textual cues
• Reveals systematic biases in semantic reasoning