DeltaMerge-LowRes: Composing Language and Task Deltas for Low-Resource Adaptation

2026-07-15

AI Summary

This work addresses the challenge of jointly adapting language models to new languages and tasks under extremely low-resource conditions—where only a few hundred labeled examples are available—by decoupling language and task adaptation. The authors propose learning separate language-specific (Δ_L) and task-specific (Δ_T) parameter increments, which are then orthogonally combined in weight space via a novel cross-axis TIES merging rule. Δ_L is trained using unlabeled monolingual data, while Δ_T leverages English-labeled data, with both updates enhanced by additive refinement, activation guidance, and sparsity-aware strategies. Evaluated on four African languages, the approach yields substantial gains: summarization chrF scores improve by 4–7 points (reaching up to 18.59), question answering F1 and exact match increase by 2.32 and 2.91 respectively, and sparse fusion reduces classification expected calibration error (ECE) by 36%.

Abstract

Adapting a multilingual encoder to a new language \emph{and} a new task with only a few hundred gold examples is a common low-resource NLP setting, yet the two axes are usually fused via an expensive language--task fine-tuning run. We ask whether they can instead be trained separately and recombined in weight space. \DeltaMergeLowRes{} learns a language delta $Δ_L$ from unlabeled monolingual text and a task delta $Δ_T$ from labeled English data, then composes them at inference under one of four rules: additive, activation-guided, sparsity-aware, and a novel \emph{cross-axis TIES}. The new rule adapts the TIES-Merging steps of trimming, sign election, and merging to the language and task axes rather than to two task axes. Holding $(Δ_L,Δ_T)$ fixed across rules on four task families and four African languages ($158$ evaluated cells, $10{,}000$-sample paired bootstrap per cell), we find: (i) cross-axis TIES wins summarisation on $3/4$ languages by $+4$ to $+7$ chrF (chrF $18.59$ vs.\ $13.80$ task-only); (ii) it improves QA F1 by $+2.32$ and EM by $+2.91$; and (iii) sparsity-aware merging cuts classification ECE by $36\%$ at parity macro-F1. The composition rule materially changes what the merged model preserves, suppresses, and calibrates. We release all JSON traces and a claim ledger.

Problem

• low-resource adaptation
• language adaptation
• task adaptation
• multilingual NLP
• few-shot learning

Innovation

• Delta Merging
• Low-Resource Adaptation
• Cross-axis TIES
• Language-Task Decomposition
• Weight Space Composition