Can MLLMs Critique Like Humans? Evaluating Open-Ended Aesthetic Reasoning in Multimodal Large Language Models

2026-06-28

AI Summary

Current multimodal large language models (MLLMs) struggle to replicate the critical expressiveness of human-generated commentary in open-ended aesthetic critique tasks, and prevailing evaluation paradigms rely heavily on numerical ratings, which fail to capture alignment between generated critiques and human-written text. This work presents the first systematic disentanglement of prompt design components—such as persona assignment, aspect guidance, and length control—leveraging the Reddit photo commentary dataset. Through image-mismatch ablation studies and a suite of multidimensional similarity metrics complemented by fine-grained behavioral analysis, we evaluate the critique quality of five open-source MLLMs. Our findings reveal that while models exhibit coarse-grained thematic consistency with human responses, they lack image-specificity and tend to produce verbose, generic, and repetitive comments, markedly diverging from the selectivity and concreteness characteristic of human critiques. Moreover, we demonstrate that conventional reference-based similarity metrics can be misleading in this context.

Abstract

Open-ended aesthetic critique is a challenge for multimodal large language models (MLLMs): unlike multiple-choice aesthetic benchmarks, it has no single correct answer, and most aesthetic evaluation has measured models against numeric scores rather than the written critiques people actually give. We evaluate MLLM critiques against ranked human references and ask whether they are close to human ones. Using the Reddit Photo Critique Dataset, we score five open-weight MLLMs against multiple ranked human critiques per photo with reference-based similarity metrics, under six prompt conditions that disentangle persona framing, aspect hinting, length control, and single- versus multi-pass generation, and add an image-grounding control that feeds each model the wrong photograph. We find that reference-based similarity gives a misleading picture. Stricter lexical and learned metrics show only weak alignment with human critiques, while a coarse embedding cosine reports broad topical overlap that the grounding control traces to a stable house style rather than image-specific observation. Behaviorally, the models diverge from humans in consistent ways the scores do not surface: even under a length cap they write two to three times as much, cover nearly every aesthetic aspect where humans are selective, engage each aspect more uniformly and at greater depth, and repeat themselves across critiques of the same photo where humans vary. We argue that reference-based similarity rewards a fluent, comprehensive critique style rather than the selectivity and specificity of human critique, and discuss implications for evaluating and training open-ended multimodal generation.

Problem

• multimodal large language models
• aesthetic critique
• open-ended generation
• human-like reasoning
• reference-based evaluation

Innovation

• multimodal large language models
• aesthetic critique
• reference-based evaluation
• image grounding control
• open-ended generation