Mixture-of-Control: State-Aware Fine-Tuning for Transformer-based Models

2026-06-30

AI Summary

Existing state-based fine-tuning methods are typically constrained to intra-block control signal updates, limiting their ability to facilitate efficient cross-block interaction; meanwhile, approaches that enable cross-block communication often incur excessive computational overhead. To address this trade-off, this work proposes the Mixture-of-Control (MoC) framework, which introduces sparse Mixture-of-Experts (Sparse MoE) mechanisms into state fine-tuning for the first time. In MoC, the control states of individual Transformer blocks are treated as experts, and a state-aware adaptive fusion mechanism enables lightweight yet effective interaction between local and global control signals. Empirical evaluations demonstrate that MoC significantly outperforms existing methods across multiple Transformer benchmark tasks while maintaining comparable memory usage and computational cost.

Abstract

State-based fine-tuning has emerged as a compelling alternative to weight-based adaptation for transformers, updating lightweight controls into states rather than model weights, offering substantial memory savings while retaining parameter efficiency. However, most existing state-based methods typically apply only per-block control updates, which limits inter-block information exchange and restricts representational adaptation. Meanwhile, prior mechanisms that enable cross-block communication often introduce considerable computational overhead, reducing their practicality for efficient fine-tuning. We introduce Mixture-of-Control (MoC), a lightweight fine-tuning framework that adaptively integrates local and global control signals to enhance representation learning. MoC treats block-wise control states as experts in a sparse mixture-of-experts process, enabling efficient communication across transformer blocks. Empirical results across diverse transformer-based benchmarks demonstrate that MoC outperforms state-based methods while maintaining a comparable memory and computational efficiency.

Problem

• state-based fine-tuning
• inter-block communication
• parameter efficiency
• transformer adaptation
• computational overhead

Innovation

• Mixture-of-Control
• state-based fine-tuning
• sparse mixture-of-experts
• cross-block communication
• parameter-efficient adaptation