Patch Rebirth: Toward Fast and Transferable Model Inversion of Vision Transformers

2025-09-27

AI Summary

To address the low efficiency and poor transferability of model inversion for Vision Transformers (ViTs) in data-free scenarios, this paper proposes a progressive sparse synthetic image generation method. Our core innovation is the Patch Rebirth mechanism: it dynamically evaluates patch importance, preserves and iteratively refines less-critical patches, thereby enabling co-evolution of class-agnostic priors and class-specific features; further, we design a progressive sparse separation and dynamic update strategy leveraging ViT’s self-attention properties. Experiments demonstrate that our method achieves 10× faster inversion than Dense MI and 2× faster than SMI, while attaining accuracy superior to SMI and comparable to DMI. It thus strikes a significant balance between inversion efficiency and cross-task generalizability, establishing a new efficient and transferable paradigm for ViT model inversion.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Web Mining and Content Analysis: Large pretrained models with web data
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Model inversion is a widely adopted technique in data-free learning that reconstructs synthetic inputs from a pretrained model through iterative optimization, without access to original training data. Unfortunately, its application to state-of-the-art Vision Transformers (ViTs) poses a major computational challenge, due to their expensive self-attention mechanisms. To address this, Sparse Model Inversion (SMI) was proposed to improve efficiency by pruning and discarding seemingly unimportant patches, which were even claimed to be obstacles to knowledge transfer. However, our empirical findings suggest the opposite: even randomly selected patches can eventually acquire transferable knowledge through continued inversion. This reveals that discarding any prematurely inverted patches is inefficient, as it suppresses the extraction of class-agnostic features essential for knowledge transfer, along with class-specific features. In this paper, we propose Patch Rebirth Inversion (PRI), a novel approach that incrementally detaches the most important patches during the inversion process to construct sparse synthetic images, while allowing the remaining patches to continue evolving for future selection. This progressive strategy not only improves efficiency, but also encourages initially less informative patches to gradually accumulate more class-relevant knowledge, a phenomenon we refer to as the Re-Birth effect, thereby effectively balancing class-agnostic and class-specific knowledge. Experimental results show that PRI achieves up to 10x faster inversion than standard Dense Model Inversion (DMI) and 2x faster than SMI, while consistently outperforming SMI in accuracy and matching the performance of DMI.

Problem

• Addresses computational inefficiency in Vision Transformer model inversion
• Improves knowledge transfer by preserving evolving patch information
• Balances class-agnostic and class-specific feature extraction during inversion

Innovation

• Incrementally detaches important patches during inversion
• Allows remaining patches to evolve for future selection
• Balances class-agnostic and class-specific knowledge efficiently