Understanding Differential Transformer Unchains Pretrained Self-Attentions

2025-05-22

AI Summary

Differential Transformers exhibit strong performance but suffer from opaque mechanisms and reliance on costly from-scratch pretraining, hindering reuse of existing pretrained weights. This work presents the first systematic dissection of differential attention’s effectiveness, identifying three key mechanisms: (i) negative attention enhances representational capacity, (ii) it reduces multi-head redundancy, and (iii) it improves learning dynamics. Building on these insights, we propose DEX—a plug-and-play, lightweight adaptation framework that integrates differential attention into arbitrary pretrained language models without full retraining. DEX achieves efficient synergy via softmax-score reuse and value-matrix differentiation. Theoretical analysis and extensive experiments demonstrate that DEX consistently boosts performance across diverse benchmarks, requiring less than 0.01% task-specific adaptation data and incurring negligible training and inference overhead.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Mixture of Experts (MoE)
• Natural Language Processing: (Large) Language Models

Application Category

• Web Mining and Content Analysis: Large pretrained models with web data
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Differential Transformer has recently gained significant attention for its impressive empirical performance, often attributed to its ability to perform noise canceled attention. However, precisely how differential attention achieves its empirical benefits remains poorly understood. Moreover, Differential Transformer architecture demands large-scale training from scratch, hindering utilization of open pretrained weights. In this work, we conduct an in-depth investigation of Differential Transformer, uncovering three key factors behind its success: (1) enhanced expressivity via negative attention, (2) reduced redundancy among attention heads, and (3) improved learning dynamics. Based on these findings, we propose DEX, a novel method to efficiently integrate the advantages of differential attention into pretrained language models. By reusing the softmax attention scores and adding a lightweight differential operation on the output value matrix, DEX effectively incorporates the key advantages of differential attention while remaining lightweight in both training and inference. Evaluations confirm that DEX substantially improves the pretrained LLMs across diverse benchmarks, achieving significant performance gains with minimal adaptation data (<0.01%).

Problem

• Understanding how Differential Transformer achieves noise canceled attention
• Enabling use of pretrained weights in Differential Transformer architecture
• Integrating differential attention benefits into pretrained models efficiently

Innovation

• Integrates differential attention into pretrained models
• Uses lightweight operation on value matrix
• Enhances performance with minimal adaptation data