Can Vision Language Models Understand Mimed Actions?

2025-06-17

Annual Meeting of the Association for Computational Linguistics

AI Summary

This study investigates visual-language models’ (VLMs’) ability to understand mimed actions—a low-ambiguity subset of nonverbal communication (NVC). To this end, we introduce MIME, the first dedicated benchmark for evaluating VLMs on mimed actions: it comprises 86 high-fidelity motion-captured video classes, systematically varying actors, backgrounds, and camera viewpoints to assess robust generalization. We adopt a video question-answering (VQA) paradigm to comprehensively evaluate both open- and closed-source VLMs. Results reveal substantial performance gaps between current VLMs and human annotators on MIME, exposing fundamental limitations in modeling gesture semantics and motion intent. This work formally establishes mimed actions as a high-robustness entry point for NVC understanding and proposes a scalable, generative evaluation framework—thereby establishing a new benchmark for embodied interaction and gesture comprehension in VLMs.

Technology Category

• Computer Vision: Video Understanding & Activity Analysis
• Machine Learning: Large Multimodal Models (LMMs)
• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior

Application Category

• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Nonverbal communication (NVC) plays an integral role in human language, but studying NVC in general is challenging because of its broad scope and high variance in interpretation among individuals and cultures. However, mime -- the theatrical technique of suggesting intent using only gesture, expression, and movement -- is a subset of NVC that consists of explicit and embodied actions with much lower human interpretation variance. We argue that a solid understanding of mimed actions is a crucial prerequisite for vision-language models capable of interpreting and commanding more subtle aspects of NVC. Hence, we propose Mime Identification Multimodal Evaluation (MIME), a novel video-based question answering benchmark comprising of 86 mimed actions. Constructed with motion capture data, MIME consists of variations of each action with perturbations applied to the character, background, and viewpoint for evaluating recognition robustness. We find that both open-weight and API-based vision-language models perform significantly worse than humans on MIME, motivating the need for increased research for instilling more robust understanding of human gestures.

Problem

• Evaluating vision-language models' understanding of mimed actions
• Assessing robustness in recognizing varied mimed action scenarios
• Addressing performance gap between models and humans in gesture comprehension

Innovation

• MIME benchmark for mimed action evaluation
• Motion capture data for action variations
• Robustness testing with character perturbations