Disentangling Regional Primitives for Image Generation

2024-10-06

arXiv.org

AI Summary

This work addresses the lack of spatial interpretability in intermediate-layer representations of generative models. We propose a feature disentanglement method based on extended Harsanyi interaction, which decomposes intermediate features into a set of region-specific primitive basis functions—each explicitly corresponding to the generation process of a distinct spatial region in the image, thereby modeling the full image as a superposition of pre-encoded regional patterns. Crucially, we introduce extended Harsanyi interaction for the first time to characterize OR-type semantic dependencies between features and multiple regions, enabling semantically clear and mutually exclusive region-feature alignment. Experiments demonstrate that each disentangled component exhibits high specificity to its target region, significantly enhancing both interpretability and spatial controllability of the generation process. Our approach establishes a novel paradigm for structured understanding and spatially grounded editing of generative models.

Technology Category

• Computer Vision: Generative Adversarial Networks (GANs) for Vision
• Machine Learning: Deep Generative Models & Autoencoders
• Natural Language Processing: Generation

Application Category

• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web query analysis, representation and understanding

Abstract

This paper presents a method to explain the internal representation structure of a neural network for image generation. Specifically, our method disentangles primitive feature components from the intermediate-layer feature of the neural network, which ensures that each feature component is exclusively used to generate a specific set of image regions. In this way, the generation of the entire image can be considered as the superposition of different pre-encoded primitive regional patterns, each being generated by a feature component. We find that the feature component can be represented as an OR relationship between the demands for generating different image regions, which is encoded by the neural network. Therefore, we extend the Harsanyi interaction to represent such an OR interaction to disentangle the feature component. Experiments show a clear correspondence between each feature component and the generation of specific image regions.

Problem

• Explaining neural network representation structures for image generation
• Disentangling primitive feature components from intermediate-layer features
• Proving feature components satisfy completeness and linear additivity properties

Innovation

• Disentangling primitive feature components from intermediate layers
• Explaining image generation as superposition of regional patterns
• Computing feature components as OR Harsanyi interaction