DeAL: Decoding-time Alignment for Large Language Models

2024-02-05

Annual Meeting of the Association for Computational Linguistics

AI Summary

Existing LLM alignment methods (e.g., RLHF) rely on static, generic principles established during training, limiting flexibility in supporting user-defined, multi-dimensional, fine-grained real-time alignment objectives—such as harmlessness, helpfulness, keyword adherence, or length constraints—and remain vulnerable to jailbreaking attacks. This paper proposes DeAL, a *decoding-time alignment* framework that shifts alignment from training to the autoregressive generation stage. DeAL integrates heuristic-constrained decoding, programmable multi-objective reward functions, and dynamically weighted scoring to enable plug-and-play, multi-objective, and real-time controllable alignment. Orthogonal to both RLHF and prompt engineering, DeAL significantly improves adherence across diverse alignment goals. Empirical results demonstrate superior robustness and jailbreak resistance in keyword control, length-constrained generation, and harmful content suppression.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Multiagent Systems: Adversarial Agents

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Large Language Models (LLMs) are nowadays expected to generate content aligned with human preferences. Current work focuses on alignment at model training time, through techniques such as Reinforcement Learning with Human Feedback (RLHF). However, it is unclear if such methods are an effective choice to teach alignment objectives to the model. First, the inability to incorporate multiple, custom rewards and reliance on a model developer's view of universal and static principles are key limitations. Second, the residual gaps in model training and the reliability of such approaches are also questionable (e.g. susceptibility to jail-breaking even after safety training). To address these, we propose DeAL, a framework that allows the user to customize reward functions and enables Decoding-time Alignment of LLMs (DeAL). At its core, we view decoding as a heuristic-guided search process and facilitate the use of a wide variety of alignment objectives. Our experiments with programmatic constraints such as keyword and length constraints (studied widely in the pre-LLM era) and abstract objectives such as harmlessness and helpfulness (proposed in the post-LLM era) show that we can DeAL with fine-grained trade-offs, improve adherence to alignment objectives, and address residual gaps in LLMs. Lastly, while DeAL can be effectively paired with RLHF and prompting techniques, its generality makes decoding slower, an optimization we leave for future work.

Problem

• Addressing limitations of training-time alignment methods
• Enabling customizable reward functions for LLM alignment
• Improving decoding-time adherence to multiple alignment objectives

Innovation

• Decoding-time alignment framework for LLMs
• Customizable reward functions for user preferences
• Heuristic-guided search for diverse alignment objectives