PPJudge: Towards Human-Aligned Assessment of Artistic Painting Process

2025-07-12

AI Summary

Existing art image evaluation methods focus solely on static outputs, neglecting the dynamic, stage-wise nature of the painting process. Method: This paper introduces the first systematic evaluation framework for painting processes, proposing PPAD—a novel eight-dimensional expert-annotated dataset comprising both authentic and synthetic procedural sequences—and PPJudge, a temporal-aware assessment model built upon a Transformer architecture. PPJudge incorporates learnable temporal positional encodings and a heterogeneous Mixture-of-Experts (MoE) module to jointly model image sequences and learn multiple artistic attributes. Contribution/Results: Experiments demonstrate that PPJudge significantly outperforms state-of-the-art methods in evaluation accuracy, robustness, and human alignment. It serves as the first interpretable, high-fidelity assessment tool explicitly designed for painting processes, advancing computational creativity research and art education.

Technology Category

• Cognitive Modeling & Cognitive Systems: Computational Creativity
• Computer Vision: Diffusion Models for Vision
• Humans and AI: Game Design — Procedural Content Generation & Storytelling

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Artistic image assessment has become a prominent research area in computer vision. In recent years, the field has witnessed a proliferation of datasets and methods designed to evaluate the aesthetic quality of paintings. However, most existing approaches focus solely on static final images, overlooking the dynamic and multi-stage nature of the artistic painting process. To address this gap, we propose a novel framework for human-aligned assessment of painting processes. Specifically, we introduce the Painting Process Assessment Dataset (PPAD), the first large-scale dataset comprising real and synthetic painting process images, annotated by domain experts across eight detailed attributes. Furthermore, we present PPJudge (Painting Process Judge), a Transformer-based model enhanced with temporally-aware positional encoding and a heterogeneous mixture-of-experts architecture, enabling effective assessment of the painting process. Experimental results demonstrate that our method outperforms existing baselines in accuracy, robustness, and alignment with human judgment, offering new insights into computational creativity and art education.

Problem

• Assessing dynamic multi-stage artistic painting processes
• Lack of datasets for painting process evaluation
• Improving accuracy and human alignment in art assessment

Innovation

• Transformer-based model with temporal encoding
• Heterogeneous mixture-of-experts architecture
• Large-scale annotated painting process dataset