A Unifying Information-theoretic Perspective on Evaluating Generative Models

2024-12-18

AI Summary

Existing evaluation metrics for generative models suffer from difficulties in jointly quantifying fidelity and diversity, strong domain dependence, and insufficient sensitivity to quality variations. Method: This paper proposes a unified, information-theoretic three-dimensional evaluation framework. It systematically integrates k-nearest-neighbor (kNN) density estimation into entropy and cross-entropy theory, yielding three orthogonal metrics—PCE (fidelity), RCE (inter-class diversity), and RE (intra-class diversity)—enabling decoupled sample-level and mode-level analysis for the first time. Results: Experiments demonstrate high sensitivity of each metric to its targeted quality dimension; expose implicit biases in mainstream metrics (e.g., Precision/Recall) regarding mode coverage and real-sample alignment; and confirm strong cross-domain generalization across diverse generative tasks. The framework establishes a theoretically consistent, interpretable, and domain-agnostic paradigm for universal generative model evaluation.

Technology Category

• Machine Learning: Deep Generative Models & Autoencoders
• Computer Vision: Generative Adversarial Networks (GANs) for Vision
• Natural Language Processing: Generation

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Considering the difficulty of interpreting generative model output, there is significant current research focused on determining meaningful evaluation metrics. Several recent approaches utilize"precision"and"recall,"borrowed from the classification domain, to individually quantify the output fidelity (realism) and output diversity (representation of the real data variation), respectively. With the increase in metric proposals, there is a need for a unifying perspective, allowing for easier comparison and clearer explanation of their benefits and drawbacks. To this end, we unify a class of kth-nearest-neighbors (kNN)-based metrics under an information-theoretic lens using approaches from kNN density estimation. Additionally, we propose a tri-dimensional metric composed of Precision Cross-Entropy (PCE), Recall Cross-Entropy (RCE), and Recall Entropy (RE), which separately measure fidelity and two distinct aspects of diversity, inter- and intra-class. Our domain-agnostic metric, derived from the information-theoretic concepts of entropy and cross-entropy, can be dissected for both sample- and mode-level analysis. Our detailed experimental results demonstrate the sensitivity of our metric components to their respective qualities and reveal undesirable behaviors of other metrics.

Problem

• Generative Model Evaluation
• Quality Assessment
• Domain-Independent Metrics

Innovation

• Unified Evaluation Framework
• Generative Models Performance
• Information-Theoretic Metrics