The Unanticipated Asymmetry Between Perceptual Optimization and Assessment

2025-09-25

AI Summary

This work identifies a critical asymmetry between image quality assessment (IQA) and perceptual optimization: high-fidelity IQA metrics (e.g., LPIPS) perform poorly in perceptual optimization tasks—especially under adversarial training—revealing a structural mismatch between evaluation and optimization objectives. Method: We systematically compare convolutional, patch-based, and Transformer-based discriminators in both generative optimization and IQA, analyzing their architectural impact on detail reconstruction, artifact suppression, and representation transferability. Contribution/Results: We find that patch-based convolutional discriminators achieve optimal perceptual optimization performance, yet their learned representations yield limited gains when transferred to IQA model backbones. This demonstrates that discriminator architecture critically governs optimization efficacy but does not produce universally transferable IQA representations. To our knowledge, this is the first empirical characterization of such an evaluation-optimization misalignment, establishing a new paradigm for discriminator design and IQA model development.

Technology Category

• Computer Vision: Generative Adversarial Networks (GANs) for Vision
• Search and Optimization: Non-convex Optimization
• Machine Learning: Optimization

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Web Mining and Content Analysis: Web data quality in the era of algorithmically-generated content

Abstract

Perceptual optimization is primarily driven by the fidelity objective, which enforces both semantic consistency and overall visual realism, while the adversarial objective provides complementary refinement by enhancing perceptual sharpness and fine-grained detail. Despite their central role, the correlation between their effectiveness as optimization objectives and their capability as image quality assessment (IQA) metrics remains underexplored. In this work, we conduct a systematic analysis and reveal an unanticipated asymmetry between perceptual optimization and assessment: fidelity metrics that excel in IQA are not necessarily effective for perceptual optimization, with this misalignment emerging more distinctly under adversarial training. In addition, while discriminators effectively suppress artifacts during optimization, their learned representations offer only limited benefits when reused as backbone initializations for IQA models. Beyond this asymmetry, our findings further demonstrate that discriminator design plays a decisive role in shaping optimization, with patch-level and convolutional architectures providing more faithful detail reconstruction than vanilla or Transformer-based alternatives. These insights advance the understanding of loss function design and its connection to IQA transferability, paving the way for more principled approaches to perceptual optimization.

Problem

• Analyzing the asymmetry between perceptual optimization and image quality assessment metrics.
• Investigating why fidelity metrics effective in IQA fail in optimization tasks.
• Evaluating how discriminator design impacts detail reconstruction during optimization.

Innovation

• Fidelity metrics misalign with optimization under adversarial training
• Discriminator designs provide more faithful detail reconstruction
• Patch-level convolutional architectures outperform Transformer-based alternatives