State-offset Tuning: State-based Parameter-Efficient Fine-Tuning for State Space Models

2025-03-05

AI Summary

This paper addresses the lack of efficient parameter-efficient fine-tuning (PEFT) methods for state space models (SSMs). Departing from conventional input-side prompting techniques (e.g., Prompt- or Prefix-Tuning), it proposes a novel, **state-centric PEFT paradigm** tailored to SSMs’ architectural characteristics. Specifically, it introduces **State-offset Tuning**, which injects lightweight, trainable state offset modules directly into the state transition step—thereby enabling precise, low-overhead adaptation by modifying the latent state variables themselves. Extensive experiments across multiple tasks and datasets demonstrate that State-offset Tuning significantly outperforms mainstream PEFT baselines: it achieves faster convergence, reduces trainable parameters by over 99%, and matches the performance of full-parameter fine-tuning.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Search and Optimization: Learning to Search
• Intelligent Robots: State Estimation

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation

Abstract

State Space Models (SSMs) have emerged as efficient alternatives to Transformers, mitigating their quadratic computational cost. However, the application of Parameter-Efficient Fine-Tuning (PEFT) methods to SSMs remains largely unexplored. In particular, prompt-based methods like Prompt Tuning and Prefix-Tuning, which are widely used in Transformers, do not perform well on SSMs. To address this, we propose state-based methods as a superior alternative to prompt-based methods. This new family of methods naturally stems from the architectural characteristics of SSMs. State-based methods adjust state-related features directly instead of depending on external prompts. Furthermore, we introduce a novel state-based PEFT method: State-offset Tuning. At every timestep, our method directly affects the state at the current step, leading to more effective adaptation. Through extensive experiments across diverse datasets, we demonstrate the effectiveness of our method. Code is available at https://github.com/furiosa-ai/ssm-state-tuning.

Problem

• Explores Parameter-Efficient Fine-Tuning for State Space Models.
• Proposes state-based methods over prompt-based for SSMs.
• Introduces State-offset Tuning for effective model adaptation.

Innovation

• State-offset Tuning for SSMs
• State-based PEFT replaces prompt-based methods
• Direct state adjustment enhances model adaptation