Tradeoffs Between Alignment and Helpfulness in Language Models

2024-01-29

arXiv.org

AI Summary

This work investigates the fundamental trade-off between alignment—encompassing adversarial robustness and bias mitigation—and helpfulness—i.e., core task performance—in language models. We propose the first provable theoretical framework that quantifies how alignment and helpfulness vary with vector norm scaling in representation engineering: alignment improves linearly, whereas helpfulness degrades quadratically. Our analysis yields tight theoretical bounds on the alignment gain versus utility loss, explicitly characterizing the effective frontier and efficiency critical point of representation engineering. Systematic empirical evaluation across adversarial robustness, social bias reduction, and general capability benchmarks validates the theoretical predictions and delineates the feasible operational window for practical deployment. The core contribution lies in rigorously uncovering and formalizing this intrinsic trade-off, thereby providing both theoretical foundations and actionable guidance for controllable alignment in large language models.

Technology Category

• Machine Learning: Adversarial Learning & Robustness
• Natural Language Processing: Safety and Robustness
• Computer Vision: Adversarial Attacks & Robustness

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Language model alignment has become an important component of AI safety, allowing safe interactions between humans and language models, by enhancing desired behaviors and inhibiting undesired ones. It is often done by tuning the model or inserting preset aligning prompts. Recently, representation engineering, a method which alters the model's behavior via changing its representations post-training, was shown to be effective in aligning LLMs (Zou et al., 2023a). Representation engineering yields gains in alignment oriented tasks such as resistance to adversarial attacks and reduction of social biases, but was also shown to cause a decrease in the ability of the model to perform basic tasks. In this paper we study the tradeoff between the increase in alignment and decrease in helpfulness of the model. We propose a theoretical framework which provides bounds for these two quantities, and demonstrate their relevance empirically. First, we find that under the conditions of our framework, alignment can be guaranteed with representation engineering, and at the same time that helpfulness is harmed in the process. Second, we show that helpfulness is harmed quadratically with the norm of the representation engineering vector, while the alignment increases linearly with it, indicating a regime in which it is efficient to use representation engineering. We validate our findings empirically, and chart the boundaries to the usefulness of representation engineering for alignment.

Problem

• Tradeoff between model alignment and helpfulness in LLMs
• Impact of representation engineering on alignment and performance
• Theoretical bounds for alignment gains and helpfulness loss

Innovation

• Uses representation engineering for model alignment
• Analyzes alignment-helpfulness tradeoff theoretically
• Demonstrates quadratic harm to helpfulness