AVMeme Exam: A Multimodal Multilingual Multicultural Benchmark for LLMs'Contextual and Cultural Knowledge and Thinking

2026-01-25

AI Summary

This study addresses the significant limitations of current AI models in comprehending the cultural context and emotional nuances of text-free audiovisual memes. To bridge this gap, the authors construct a multimodal benchmark comprising over a thousand iconic audiovisual memes—including speech, songs, music, and sound effects—curated with human annotation and enriched with structured metadata such as year, transcription, summary, and sensitivity tags. The work introduces a hierarchical question-answering framework that progresses from surface-level content to deep cultural interpretation. Notably, it presents the first systematic evaluation framework integrating cultural, contextual, and affective dimensions into multimodal meme understanding, supporting fine-grained assessment across languages, cultures, and modalities. Experimental results reveal that state-of-the-art multimodal large language models substantially underperform humans in text-free audio comprehension and cultural reasoning, highlighting a critical gap in cultural alignment.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Computer Vision: Multi-modal Vision

Application Category

• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling

Abstract

Internet audio-visual clips convey meaning through time-varying sound and motion, which extend beyond what text alone can represent. To examine whether AI models can understand such signals in human cultural contexts, we introduce AVMeme Exam, a human-curated benchmark of over one thousand iconic Internet sounds and videos spanning speech, songs, music, and sound effects. Each meme is paired with a unique Q&A assessing levels of understanding from surface content to context and emotion to usage and world knowledge, along with metadata such as original year, transcript, summary, and sensitivity. We systematically evaluate state-of-the-art multimodal large language models (MLLMs) alongside human participants using this benchmark. Our results reveal a consistent limitation: current models perform poorly on textless music and sound effects, and struggle to think in context and in culture compared to surface content. These findings highlight a key gap in human-aligned multimodal intelligence and call for models that can perceive contextually and culturally beyond the surface of what they hear and see. Project page: avmemeexam.github.io/public

Problem

• multimodal
• cultural understanding
• contextual reasoning
• audio-visual memes
• large language models

Innovation

• multimodal benchmark
• cultural understanding
• audio-visual memes
• contextual reasoning
• multilingual evaluation