Rare Concept Generation via Counterfactual Inference in Diffusion Models

2026-07-16

AI Summary

This work addresses the challenge that diffusion models often produce attribute errors or inconsistent compositions when generating images of rare concepts due to commonsense biases present in training data. To mitigate this issue, the authors propose CI-Diff, a novel approach that introduces causal inference into rare concept generation for the first time. By leveraging counterfactual reasoning, CI-Diff blocks the influence of commonsense biases and employs the natural direct effect to disentangle the independent contributions of rare concepts and their unconventional attributes in text prompts. Furthermore, it reformulates the classifier-free guidance mechanism to enhance the expression of such unconventional attributes. Experimental results on the RareBench benchmark demonstrate that CI-Diff significantly outperforms state-of-the-art diffusion models, markedly improving both the accuracy and consistency of generated images.

Abstract

Rare concept generation focuses on synthesizing customized images conditioned on text prompts that describe objects with unusual attributes. Previous works failed to align the generated images with rare concepts, resulting in incorrect attribute rendering or inconsistent composition of concepts. Such failures, as we observed, stem from the inherent common knowledge bias in the training stage of diffusion models, where objects are strongly associated with their common attributes, making it difficult to break these associations when generating rare concepts. To address such challenges, in this paper, we propose a novel Counterfactual Inference-based Diffusion approach, dubbed CI-Diff. CI-Diff blocks the interference of the model's inherent common knowledge bias and utilizes the Natural Direct Effect to capture the independent influence of the text prompt of rare concepts on image generation so that decoupling the unusual attributes from the rare concepts. To this end, we reformulate the classifier-free guidance mechanism to highlight the atypical attributes. To the best of our knowledge, we are the first to introduce causal inference into the rare concept generation task. Extensive experiments on the RareBench benchmark validate the superiority of CI-Diff over state-of-the-art diffusion models. Our code can be accessed from https://github.com/200204jzy/CI-Diff.

Problem

• Rare Concept Generation
• Common Knowledge Bias
• Diffusion Models
• Attribute Rendering
• Concept Composition

Innovation

• Counterfactual Inference
• Diffusion Models
• Rare Concept Generation
• Causal Inference
• Natural Direct Effect