Identification of Novel Modes in Generative Models via Fourier-based Differential Clustering

2024-05-04

AI Summary

Existing evaluation of generative models relies heavily on quantitative metrics, lacking interpretable, semantic-level comparisons across models. Method: We propose FINC, a differentiable clustering framework that integrates random Fourier features with principal eigenvectors of the kernel cross-covariance matrix to identify and localize fine-grained semantic discrepancies in sample-type distributions across models; Monte Carlo sampling enables efficient discovery of differential patterns. Contribution/Results: Evaluated on large-scale visual datasets, FINC accurately localizes semantically distinct sample clusters—e.g., revealing systematic disparities between GANs and diffusion models—enabling fine-grained, attribution-aware analysis of generative behavior. The method is scalable, interpretable, and provides a novel paradigm for diagnosing and improving generative models through semantic differential analysis.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Deep Generative Models & Autoencoders
• Natural Language Processing: Generation

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Large pretrained models with web data
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

An interpretable comparison of generative models requires the identification of sample types produced more frequently by each of the involved models. While several quantitative scores have been proposed in the literature to rank different generative models, such score-based evaluations do not reveal the nuanced differences between the generative models in capturing various sample types. In this work, we attempt to solve a differential clustering problem to detect sample types expressed differently by two generative models. To solve the differential clustering problem, we propose a method called Fourier-based Identification of Novel Clusters (FINC) to identify modes produced by a generative model with a higher frequency in comparison to a reference distribution. FINC provides a scalable stochastic algorithm based on random Fourier features to estimate the eigenspace of kernel covariance matrices of two generative models and utilize the principal eigendirections to detect the sample types present more dominantly in each model. We demonstrate the application of the FINC method to large-scale computer vision datasets and generative model frameworks. Our numerical results suggest the scalability of the developed Fourier-based method in highlighting the sample types produced with different frequencies by widely-used generative models. Code is available at url{https://github.com/buyeah1109/FINC}

Problem

• Identify sample types generated differently by models
• Detect nuanced differences between generative models
• Develop scalable differential clustering approach FINC

Innovation

• Spectral method FINC for differential clustering
• Random Fourier features for scalable eigenspace estimation
• Principal eigendirections detect dominant sample types