SHIFT: Steering Hidden Intermediates in Flow Transformers

2026-04-10

AI Summary

This work proposes a training-free, inference-time intervention method for efficiently removing undesirable visual concepts or steering generation in Diffusion Transformers (DiTs). Inspired by activation steering techniques in large language models, the approach introduces, for the first time, a lightweight and flexible control mechanism into the DiT architecture by dynamically injecting steering vectors—derived from intermediate activations—at specific layers and timesteps during the diffusion process. Experimental results demonstrate that the method effectively suppresses targeted concepts or modulates style and object attributes across diverse prompts and objectives, while preserving overall image quality and semantic consistency.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Active Learning
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning

Application Category

• Web Mining and Content Analysis: Large pretrained models with web data
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Diffusion models have become leading approaches for high-fidelity image generation. Recent DiT-based diffusion models, in particular, achieve strong prompt adherence while producing high-quality samples. We propose SHIFT, a simple but effective and lightweight framework for concept removal in DiT diffusion models via targeted manipulation of intermediate activations at inference time, inspired by activation steering in large language models. SHIFT learns steering vectors that are dynamically applied to selected layers and timesteps to suppress unwanted visual concepts while preserving the prompt's remaining content and overall image quality. Beyond suppression, the same mechanism can shift generations into a desired \emph{style domain} or bias samples toward adding or changing target objects. We demonstrate that SHIFT provides effective and flexible control over DiT generation across diverse prompts and targets without time-consuming retraining.

Problem

• concept removal
• diffusion models
• activation steering
• image generation
• style control

Innovation

• activation steering
• concept removal
• diffusion transformers
• inference-time intervention
• style control