Evaluating Multimodal Narrative Understanding of Popular Hollywood Films

2026-08-17

AI Summary

本文通过构建一个基于公共领域好莱坞电影的多模态选择题基准，解决版权问题，并评估模型对电影叙事的理解能力。

Abstract

Multimodal language models increasingly show promise for enabling the large-scale computational analysis of film, opening up new avenues for learning about film history and the evolution of narrative techniques. But the creation of stable benchmarks built around Hollywood films is complicated by copyright protections. In this work, we address these concerns directly, by building a new collection of Hollywood films defined by two criteria: box office popularity (where we publish the first large-scale, open collection of weekly box office earnings reported by Variety magazine from 1922-1979); and likely public domain status (by researching copyright registrations and renewals in the US Catalog of Copyright Entries). We build a new multimodal MCQ benchmark on top of this collection that focuses on narrative elements that directly evaluate the abilities of models to inform meaningful research on film narrative; we find that many vision-language models struggle on this task (with many performing at near-chance levels of accuracy), while audio-visual models (including those that use audio in captioning scenes) reach a maximum accuracy of 61.1%, well below human-level performance.

Problem

• Multimodal Narrative Understanding
• Hollywood Films
• Benchmarking
• Copyright Protections

Innovation

• multimodal MCQ benchmark
• public domain status
• audio-visual models