VT-DUDA: Visual Token Conditioning for Diffusion-guided Unsupervised Domain Adaptation

2026-06-19

AI Summary

This work addresses the performance degradation in unsupervised domain adaptation (UDA) caused by distribution shifts in the target domain and the coarse conditional guidance of existing diffusion models for style synthesis. To this end, the authors propose a vision-token-based hybrid conditional guidance framework that, for the first time, incorporates instance-level visual tokens into diffusion models. These tokens are concatenated with text embeddings to form a joint conditional input, enabling fine-grained and controllable generation of target-domain images through cross-attention mechanisms. Notably, the approach requires no modification to the backbone network and leverages only a domain adapter branch alongside a latent diffusion model to produce high-fidelity synthetic data. Experiments on Office-31, Office-Home, and VisDA-2017 benchmarks demonstrate substantial improvements over current discriminative and diffusion-based UDA methods, significantly boosting average classification accuracy on the target domain.

Abstract

Unsupervised domain adaptation (UDA) aims to learn a target-domain classifier from labeled source data and unlabeled target data under distribution shift. Recent diffusion-based UDA methods approach this problem by synthesizing labeled target-style images and training on the resulting synthetic data. However, their performance depends heavily on the conditioning design: class prompts provide only coarse guidance, while domain adaptation modules mainly control appearance, which may leave target-style synthesis insufficiently specified. We propose VT-DUDA, a visual-token conditioning framework for diffusion-guided UDA. Instead of relying only on text prompts, VT-DUDA uses source images to provide additional instance-level visual context for target-style synthesis. Specifically, VT-DUDA maps each source image to a compact sequence of visual tokens and forms a hybrid conditioning context by concatenating these tokens with the corresponding text embeddings along the cross-attention context dimension of a latent diffusion model. This provides instance-dependent conditioning beyond text alone, while synthesis is performed with the target-domain adapter branch. Because guidance is represented explicitly as a token sequence, the same interface also permits inference-time manipulation of the conditioning signal through token selection and token-strength adjustment. The proposed method preserves the standard diffusion objective and can be integrated into existing adapter-based diffusion frameworks without modifying the backbone. Across Office-31, Office-Home, and VisDA-2017, VT-DUDA improves average target-domain accuracy over strong discriminative and diffusion-based UDA baselines. The results suggest that, in generation-based UDA, a stronger conditioning interface can improve the downstream usefulness of synthetic target-style data.

Problem

• Unsupervised Domain Adaptation
• Diffusion Models
• Target-style Synthesis
• Conditioning Design
• Distribution Shift

Innovation

• visual token conditioning
• diffusion-guided UDA
• instance-level guidance
• hybrid conditioning
• latent diffusion model