Hyper-Transforming Latent Diffusion Models

2025-04-23

AI Summary

This work addresses the limited representational capacity and poor scalability of existing generative models for function synthesis, which rely on MLP-based hypernetworks. We propose a novel framework integrating implicit neural representations (INRs) with latent diffusion models (LDMs). Our method introduces two key innovations: (1) the first use of a Transformer as a hypernetwork embedded within the LDM decoder to directly synthesize INR parameters from latent variables; and (2) a “hyper-transformation” strategy that enables zero-shot adaptation of pre-trained LDMs to INR generation via minimal decoder fine-tuning—eliminating the need for full-model retraining. Evaluated on image and 3D function modeling tasks, our approach achieves substantial improvements in reconstruction fidelity and out-of-distribution generalization. Moreover, it attains significantly higher training efficiency compared to MLP-based hypernetworks, delivering both superior representational power and computational efficiency.

Technology Category

• Machine Learning: Deep Generative Models & Autoencoders
• Computer Vision: Diffusion Models for Vision
• Natural Language Processing: Generation

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

We introduce a novel generative framework for functions by integrating Implicit Neural Representations (INRs) and Transformer-based hypernetworks into latent variable models. Unlike prior approaches that rely on MLP-based hypernetworks with scalability limitations, our method employs a Transformer-based decoder to generate INR parameters from latent variables, addressing both representation capacity and computational efficiency. Our framework extends latent diffusion models (LDMs) to INR generation by replacing standard decoders with a Transformer-based hypernetwork, which can be trained either from scratch or via hyper-transforming-a strategy that fine-tunes only the decoder while freezing the pre-trained latent space. This enables efficient adaptation of existing generative models to INR-based representations without requiring full retraining.

Problem

• Develop a generative framework for functions using INRs and Transformers
• Address scalability and efficiency in latent variable models
• Enable adaptation of generative models to INR representations

Innovation

• Transformer-based hypernetwork generates INR parameters
• Extends latent diffusion models with Transformer decoder
• Enables efficient adaptation via hyper-transforming strategy