Teamwork: Collaborative Diffusion with Low-rank Coordination and Adaptation

2025-10-06

AI Summary

Existing channel-expansion methods for diffusion models in graphics are task-specific, lacking generality across models or adaptability to new tasks. Method: We propose a universal channel-expansion framework based on multi-instance collaboration and Low-Rank Adaptation (LoRA). It dynamically orchestrates multiple pre-trained diffusion models as “teammates,” jointly modeling input/output channel expansion and inter-model collaboration, enabling parameter-efficient fine-tuning and runtime activation/deactivation. Contribution/Results: By innovatively integrating LoRA into multi-model collaborative training—without altering original model architectures—our framework generalizes across diverse generative and inverse graphics tasks, including neural rendering, SVBRDF estimation, intrinsic image decomposition, and image inpainting. Experiments demonstrate significant improvements in cross-task transferability and performance robustness while maintaining model lightweightness.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Cognitive Modeling & Cognitive Systems: Computational Creativity

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG

Abstract

Large pretrained diffusion models can provide strong priors beneficial for many graphics applications. However, generative applications such as neural rendering and inverse methods such as SVBRDF estimation and intrinsic image decomposition require additional input or output channels. Current solutions for channel expansion are often application specific and these solutions can be difficult to adapt to different diffusion models or new tasks. This paper introduces Teamwork: a flexible and efficient unified solution for jointly increasing the number of input and output channels as well as adapting a pretrained diffusion model to new tasks. Teamwork achieves channel expansion without altering the pretrained diffusion model architecture by coordinating and adapting multiple instances of the base diffusion model (ie, teammates). We employ a novel variation of Low Rank-Adaptation (LoRA) to jointly address both adaptation and coordination between the different teammates. Furthermore Teamwork supports dynamic (de)activation of teammates. We demonstrate the flexibility and efficiency of Teamwork on a variety of generative and inverse graphics tasks such as inpainting, single image SVBRDF estimation, intrinsic decomposition, neural shading, and intrinsic image synthesis.

Problem

• Expanding input/output channels for diffusion models
• Adapting pretrained models to new graphics tasks
• Coordinating multiple model instances without architectural changes

Innovation

• Coordinates multiple diffusion models for expansion
• Uses Low Rank-Adaptation for model coordination
• Enables dynamic activation of model components