Toward Stable Value Alignment: Introducing Independent Modules for Consistent Value Guidance

2026-05-12

AI Summary

This work addresses the challenge that human value representations in large language models are susceptible to the dynamic nature of residual stream activations, leading to inconsistent value alignment across contexts. To mitigate this, the authors propose the Stable Value-Guided Transformer (SVGT), which incorporates a decoupled, dedicated value module to construct normative representations and introduces learnable bridge tokens as dynamic value anchors. These components explicitly steer the generation trajectory without interfering with the backbone network. Evaluated across multiple backbone architectures and safety benchmarks, SVGT reduces harmful output scores by over 70% while preserving text fluency, demonstrating the efficacy of structured value modeling in achieving robust and consistent value alignment.

Abstract

Aligning large language models (LLMs) with human values typically relies on post-training or inference-time steering that directly manipulates the backbone's parameters or representation space. However, a critical gap exists: the model's residual stream is highly dynamic, in which values exist as fragile, low-dimensional properties, inherently incompatible with the stability required for consistent value expression. In this paper, we propose the Stable Value Guidance Transformer (SVGT), which addresses this gap through an independent value module incorporating two key designs: (1) independent value modeling, maintaining normative representations in a dedicated value space isolated from the backbone, and (2) explicit behavioral guidance, transducing these stable signals into learnable latent Bridge Tokens. These tokens serve as dynamic value anchors to explicitly steer the generative trajectory, ensuring robust adherence across diverse contexts without disrupting the backbone's internal representations. Experiments across multiple backbones and safety benchmarks show that SVGT generally reduces harmful scores by over 70% while maintaining generation fluency, demonstrating the efficacy of architecturally grounded value modeling. Our code is available at https://github.com/Clervils/SVGT.git.

Problem

• value alignment
• large language models
• stable value expression
• residual stream
• dynamic representations

Innovation

• Stable Value Alignment
• Independent Value Module
• Bridge Tokens
• Value Guidance
• Large Language Models