Attention, May I Have Your Decision? Localizing Generative Choices in Diffusion Models

2026-03-29

AI Summary

Text-to-image diffusion models must make implicit generative decisions when faced with incomplete prompts, yet the internal mechanisms underlying these choices remain poorly understood. This work proposes a localization method based on attribute disentanglement probing and reveals, for the first time, that such implicit decisions are primarily governed by self-attention layers. Building on this insight, the authors develop an Implicit Choice Modification (ICM) intervention strategy that enables precise guidance of image generation by modulating only a few critical layers. The approach substantially outperforms current state-of-the-art methods in debiasing tasks while effectively reducing visual artifacts, achieving efficient and minimally intrusive model control.

Abstract

Text-to-image diffusion models exhibit remarkable generative capabilities, yet their internal operations remain opaque, particularly when handling prompts that are not fully descriptive. In such scenarios, models must make implicit decisions to generate details not explicitly specified in the text. This work investigates the hypothesis that this decision-making process is not diffuse but is computationally localized within the model's architecture. While existing localization techniques focus on prompt-related interventions, we notice that such explicit conditioning may differ from implicit decisions. Therefore, we introduce a probing-based localization technique to identify the layers with the highest attribute separability for concepts. Our findings indicate that the resolution of ambiguous concepts is governed principally by self-attention layers, identifying them as the most effective point for intervention. Based on this discovery, we propose ICM (Implicit Choice-Modification) - a precise steering method that applies targeted interventions to a small subset of layers. Extensive experiments confirm that intervening on these specific self-attention layers yields superior debiasing performance compared to existing state-of-the-art methods, minimizing artifacts common to less precise approaches. The code is available at https://github.com/kzaleskaa/icm.

Problem

• diffusion models
• implicit decisions
• localization
• self-attention
• text-to-image generation

Innovation

• diffusion models
• self-attention layers
• implicit decision-making
• probing-based localization
• ICM