MuseBench: Benchmarking Intent-Level Audiovisual Arts Understanding in MLLMs

2026-06-29

AI Summary

This study addresses a critical gap in the evaluation of multimodal large language models (MLLMs), which predominantly focus on perceptual recognition while neglecting deeper understanding of creative intent in audiovisual arts. To bridge this gap, the authors introduce the first comprehensive benchmark encompassing film, static visual art, stage performance, and game art, featuring 4,016 open-ended questions combining single- and multiple-choice formats. Data quality and challenge are ensured through a rigorous four-stage pipeline involving video essay distillation, shortcut-path filtering, adversarial distractor generation, and expert validation. Zero-shot evaluations across 28 state-of-the-art MLLMs reveal that even the best-performing model achieves only 48.29% accuracy—substantially lower than the 87.18% attained by human experts—highlighting a fundamental deficiency in current models’ capacity to interpret artistic intentionality.

Abstract

Audiovisual arts encompass diverse creative disciplines, including cinema, visual arts, stage performance, and game design, where artistic meaning arises from deliberate combinations of visual, auditory, and narrative elements (e.g., fear amplified through claustrophobic framing, or grief conveyed through silence and lingering close-ups). True artistic understanding extends beyond recognizing what is depicted to reasoning about why it is expressed through particular creative choices. Despite the strong progress of multimodal large language models (MLLMs), this critical aspect of artistic understanding remains underexplored, as existing benchmarks largely measure perceptual recognition while overlooking reasoning about creative intent. To address this gap, we introduce Musebench, a comprehensive benchmark designed to evaluate MLLMs on nuanced artistic understanding. It comprises 4,016 questions spanning cinematic arts, static visual arts, stage performing arts, and game arts, distilled from over 10K candidate video essays that pair professional commentary with visual demonstration. To capture the open-ended nature of artistic analysis at scale, the benchmark combines single-select and variable-option multi-select questions. All questions are generated and refined through a four-phase iterative pipeline combining shortcut filtering, adversarial distractors, and expert validation. Comprehensive zero-shot evaluation of 28 state-of-the-art MLLMs reveals that even the best-performing model achieves only 48.29% accuracy, substantially below human expert performance of 87.18%, exposing a significant gap in current models' creative domain expertise.

Problem

• artistic understanding
• creative intent
• multimodal large language models
• audiovisual arts
• benchmarking

Innovation

• artistic intent understanding
• multimodal large language models
• audiovisual arts benchmark
• creative reasoning
• MuseBench