Co-Adaptive Multi-Task LoRA: Transfer-Aware, Label-Free Control of Domain Participation

2026-07-03

AI Summary

This work addresses the limitations of negative transfer across domains and static mixing strategies in multi-task low-rank adaptation (LoRA) by proposing CoDA, an unsupervised collaborative adaptive controller. CoDA dynamically assesses each domain’s learning potential and cross-domain affinity through unsupervised probes during forward passes, and jointly optimizes data sampling ratios and loss weights via a transfer-aware water-filling strategy. Theoretical analysis establishes its convergence guarantees. Experimental results demonstrate that, using only half the training data, CoDA significantly outperforms baseline approaches—including uniform mixing, learnable mixing, and gradient surgery—across five heterogeneous domains and two backbone architectures, effectively mitigating gradient conflicts and improving average performance.

Abstract

Fine-tuning a single low-rank adapter on many domains at once is multi-task learning: the domains must be co-learned, and how they share the adapter decides whether they help or hurt one another. Most efficient fine-tuning pipelines ignore this and train on a fixed, uniform mixture, leaving two coupled questions unanswered: how much should each domain participate, and which domains should be co-trained given that some transfer positively and others interfere? We show that both answers can be read off cheaply and without labels. A forward pass of the current shared adapter over a small unlabeled probe yields, per domain, a competence signal whose level tracks remaining headroom and whose trajectory tracks learning speed; the drift of these probe representations yields a signed cross-domain affinity that predicts pairwise transfer. We fold both into CoDA, a co-adaptive controller that solves a small entropy-regularized quadratic program on the simplex to set each domain's participation -- jointly its loss weight and its share of the sampled data -- rewarding high-headroom, still-learning, mutually synergistic domains and damping interfering ones. The controller is forward-only, adds no trainable parameters, and wraps any multi-task LoRA pipeline. Across five heterogeneous domains and two backbones, CoDA improves the average over uniform mixing, learned mixtures, gradient-surgery multi-task optimizers, and online data selection while using half the data, and lowers cross-domain gradient conflict. We prove that the competence signal tracks domain risk, that the participation program has a unique fixed point reached by a contraction, and that its solution performs transfer-aware water-filling; analysis, ablations, and controls corroborate each claim.

Problem

• multi-task learning
• domain adaptation
• low-rank adaptation
• transfer learning
• co-training

Innovation

• Co-Adaptive Control
• Label-Free Multi-Task Learning
• Transfer-Aware Optimization
• Low-Rank Adaptation (LoRA)
• Domain Affinity Estimation