FiLM-Coordinated Dual-Branch Transformer for Global-Local Dependency Modeling in Language Modeling

2026-06-18

AI Summary

Standard Transformers model both global dependencies and local patterns within a single self-attention pathway, often leading to conflicts between long-range reasoning and fine-grained representation. To address this, this work proposes a dual-branch Transformer architecture that explicitly separates global and local modeling at each layer and introduces a bidirectional feature-wise linear modulation (FiLM) mechanism. In this mechanism, the two branches dynamically generate channel-wise scaling and shifting parameters for each other, enabling input-dependent, layer-specific, and channel-selective coordination. This lightweight channel calibration replaces computationally expensive token-level interactions. Experiments on TinyShakespeare and a WikiText-2 subset demonstrate that the proposed method significantly outperforms both single-branch and ablated dual-branch baselines under identical model width, with consistent gains verified across multiple random seeds.

Abstract

Standard Transformers use a single self-attention pathway to model both global dependencies and local patterns, creating tension between long-range structural reasoning and fine-grained local representation learning. We propose a FiLM-coordinated dual-branch Transformer for language modeling, where each layer explicitly contains a global branch and a local branch, and feature-wise linear modulation (FiLM) is used for dynamic cross-branch coordination instead of simple concatenation or static addition. The key idea is that the two branches represent different dependency views of the same input, making channel-wise calibration more suitable than heavy token-level interaction. We therefore design a bidirectional FiLM module in which each branch generates per-channel scaling and shifting parameters to condition the other. Experiments on multiple small-scale language modeling settings show that the proposed structure consistently outperforms same-width single-branch baselines and weakened dual-branch variants under a fixed lightweight configuration. On TinyShakespeare and a 1M-character subset of WikiText-2, the full dual-branch FiLM model achieves the best results among same-width structural baselines. Multi-seed results support the stability of the gains, while mechanistic analyses show that FiLM learns input-dependent, layer-dependent, and channel-selective modulation patterns rather than static scaling. Parameter-matched widened single-branch baselines also indicate that the current design still leaves room for improvement in parameter efficiency.

Problem

• language modeling
• global-local dependency
• Transformer
• self-attention
• representation learning

Innovation

• FiLM
• dual-branch Transformer
• global-local dependency
• feature-wise linear modulation
• language modeling