Problem
Research questions and friction points this paper is trying to address.
Text-to-image generation
Data efficiency
Classification diversity
Discriminative Class Tokens for Text-to-Image Diffusion Models
🤖 AI Summary
To address low class accuracy in fine-grained semantic generation, heavy reliance on labeled data, and difficulty integrating classification priors in text-to-image diffusion models, this paper proposes a non-intrusive token-level discriminative guidance method. It introduces a single learnable discriminative class token into the text embedding and iteratively optimizes it using gradient signals from a frozen pre-trained classifier—without modifying the diffusion model architecture, retraining the diffusion model, requiring labeled images, or fine-tuning the classifier. This enables end-to-end alignment between text prompts and classification priors. The method significantly improves class accuracy of generated images, facilitates efficient data augmentation in low-resource settings, and exposes biases inherent in the classifier’s training data. Inference is lightweight and sample-free, requiring no additional image inputs.
📝 Abstract
Recent advances in text-to-image diffusion models have enabled the generation of diverse and high-quality images. While impressive, the images often fall short of depicting subtle details and are susceptible to errors due to ambiguity in the input text. One way of alleviating these issues is to train diffusion models on class-labeled datasets. This approach has two disadvantages: (i) supervised datasets are generally small compared to large-scale scraped text-image datasets on which text-to-image models are trained, affecting the quality and diversity of the generated images, or (ii) the input is a hard-coded label, as opposed to free-form text, limiting the control over the generated images.In this work, we propose a non-invasive fine-tuning technique that capitalizes on the expressive potential of freeform text while achieving high accuracy through discriminative signals from a pretrained classifier. This is done by iteratively modifying the embedding of an added input token of a text-to-image diffusion model, by steering generated images toward a given target class according to a classifier. Our method is fast compared to prior fine-tuning methods and does not require a collection of in-class images or retraining of a noise-tolerant classifier. We evaluate our method extensively, showing that the generated images are: (i) more accurate and of higher quality than standard diffusion models, (ii) can be used to augment training data in a low-resource setting, and (iii) reveal information about the data used to train the guiding classifier. The code is available at https://github.com/idansc/discriminative_class_tokens.
Innovation
Methods, ideas, or system contributions that make the work stand out.
Pre-trained Classifier
Text-to-Image Generation
Data Augmentation
🔎 Similar Papers
No similar papers found.
