Artifact-Bench: Evaluating MLLMs on Detecting and Assessing the Artifacts of AI-Generated Videos

2026-05-18

AI Summary

This work addresses the lack of systematic evaluation of multimodal large language models (MLLMs) in perceiving and diagnosing distortion artifacts in AI-generated videos, particularly those in non-photorealistic styles such as animation and computer graphics. To bridge this gap, the authors introduce the first benchmark specifically designed for assessing MLLMs’ capabilities in AI video artifact detection. Built upon a three-tier artifact taxonomy, the benchmark encompasses three tasks: authenticity discrimination, pairwise comparison, and fine-grained artifact identification, complemented by human perceptual preference data for comprehensive evaluation. Experiments across 19 prominent MLLMs reveal that current models exhibit substantially limited performance in artifact recognition—often near or below random chance in certain scenarios—and display notable discrepancies from human judgments, underscoring both the necessity and challenges of advancing research in this domain.

Abstract

Recent video generative models have greatly improved the realism of AI-generated videos, yet their outputs still exhibit artifacts such as temporal inconsistencies, structural distortions, and semantic incoherence. While Multimodal Large Language Models (MLLMs) show strong visual understanding capabilities, their ability to perceive and reason about such artifacts remains unclear. Existing benchmarks often lack systematic evaluation of artifact-aware perception and fine-grained diagnostic reasoning, especially across diverse AI-generated video domains beyond photorealistic content. To address this gap, we introduce Artifact-Bench, a comprehensive benchmark for evaluating MLLMs on AI-generated video artifact detection and analysis. We first establish a three-level hierarchical taxonomy of realism artifacts, covering photorealistic, animated, and CG-style videos. Based on this taxonomy, Artifact-Bench defines three complementary tasks: real vs. AI-generated video classification, pairwise realism comparison, and fine-grained artifact identification. Experiments on 19 leading MLLMs reveal substantial limitations in artifact perception and reasoning, with many models approaching random or even below-random performance in challenging settings. We further observe significant misalignment between MLLM judgments and human perceptual preferences, highlighting their limited reliability as general evaluators for AI-generated video realism.

Problem

• AI-generated videos
• artifacts
• Multimodal Large Language Models
• video realism
• evaluation benchmark

Innovation

• Artifact-Bench
• AI-generated videos
• Multimodal Large Language Models
• artifact taxonomy
• realism evaluation