RelationAdapter: Learning and Transferring Visual Relation with Diffusion Transformers

2025-06-03

AI Summary

Existing single-reference image editing methods struggle to model and transfer non-rigid, content-aware visual relationships. To address this, we propose the first few-shot visual relation editing framework designed for editing intent generalization. First, we introduce RelationAdapter—a lightweight module that, for the first time, integrates explicit relational modeling into the Diffusion Transformer (DiT) architecture, enabling context-aware extraction and transfer of editing intents. Second, we construct Relation252K, a large-scale benchmark encompassing 218 distinct relational editing tasks, thereby filling a critical gap in evaluation resources for this domain. Extensive experiments demonstrate that our method significantly outperforms single-reference baselines in editing accuracy, generation quality, and cross-image intent transfer capability. This work establishes a new paradigm for visual editing—shifting the focus from superficial appearance adjustment toward deep semantic relationship understanding.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Reasoning under Uncertainty: Relational Probabilistic Models
• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG

Abstract

Inspired by the in-context learning mechanism of large language models (LLMs), a new paradigm of generalizable visual prompt-based image editing is emerging. Existing single-reference methods typically focus on style or appearance adjustments and struggle with non-rigid transformations. To address these limitations, we propose leveraging source-target image pairs to extract and transfer content-aware editing intent to novel query images. To this end, we introduce RelationAdapter, a lightweight module that enables Diffusion Transformer (DiT) based models to effectively capture and apply visual transformations from minimal examples. We also introduce Relation252K, a comprehensive dataset comprising 218 diverse editing tasks, to evaluate model generalization and adaptability in visual prompt-driven scenarios. Experiments on Relation252K show that RelationAdapter significantly improves the model's ability to understand and transfer editing intent, leading to notable gains in generation quality and overall editing performance.

Problem

• Enabling non-rigid image transformations via visual prompts
• Transferring editing intent from source-target pairs to new images
• Improving generalization in visual prompt-driven editing tasks

Innovation

• Leverages source-target pairs for content-aware editing
• Introduces RelationAdapter for DiT-based transformation learning
• Uses Relation252K dataset for evaluating model generalization