OrthoTryOn: Geometric Orthogonalization for Conflict-Free Unified Fashion Generation

2026-06-26

AI Summary

This work addresses gradient conflict and negative transfer in unified fashion generation caused by semantic discrepancies across tasks. To mitigate these issues, the authors propose Orthogonal Subspace Projection (OSP) and Fisher-guided Negative Guidance (FNG). OSP constructs task-specific orthogonal subspaces within a shared LoRA module, decoupling bottleneck features into independent coordinate systems. During inference, FNG leverages Fisher information to suppress cross-task semantic interference. This approach is the first to effectively eliminate gradient conflicts within a unified diffusion framework, significantly enhancing generalization. It outperforms task-specific models on multiple benchmarks and demonstrates strong robustness across diverse backbone architectures.

Abstract

Unified fashion generation integrates tasks like virtual try-on and garment reconstruction into a single model to reduce task-specific adaptation costs. However, naive parameter sharing across semantically distinct tasks induces negative transfer through severe inter-task gradient conflict. We propose OrthoTryOn, a unified framework mitigating this interference within a shared Low-Rank Adaptation (LoRA) module. Its Orthogonal Subspace Projection (OSP) applies task-specific orthogonal rotations to bottleneck features, mapping them into decorrelated coordinate frames. To address residual semantic coupling at inference time, we further propose Fisher-guided Negative Guidance (FNG), a parameter-free strategy that utilizes diagonal Fisher information to quantify inter-task sensitivity overlap and explicitly repels generation trajectories from the most confusable task via Classifier-Free Guidance. Extensive experiments demonstrate that OrthoTryOn avoids the severe performance degradation typical of naive unified training and even surpasses independently trained task-specific models, achieving state-of-the-art results across multiple benchmarks while generalizing robustly across diverse diffusion backbones. Code is available at https://github.com/NJU-PCALab/OrthoTryOn.

Problem

• unified fashion generation
• gradient conflict
• negative transfer
• task interference
• parameter sharing

Innovation

• Orthogonal Subspace Projection
• Fisher-guided Negative Guidance
• Low-Rank Adaptation
• Unified Fashion Generation
• Gradient Conflict Mitigation