SPRI: Aligning Large Language Models with Context-Situated Principles

2025-02-05

AI Summary

To address the challenge of dynamically aligning large language models (LLMs) with human values in complex tasks, this paper proposes SPRI—a framework leveraging meta-reasoning and principle synthesis to generate contextualized, fine-grained guidance principles for each input query, enabling lightweight, automatic, instance-level value alignment. SPRI eliminates reliance on handcrafted rules by dynamically guiding response generation, synthesizing supervised fine-tuning (SFT) data, and constructing principle-driven, instance-specific evaluation rubrics—thereby overcoming the generalization bottleneck of universal principles. Experiments demonstrate that SPRI matches expert-designed principles in domain-specific tasks; its automatically generated evaluation rubrics significantly outperform existing LLM-as-a-judge approaches; and the synthesized SFT data substantially improves model response veracity and value consistency.

Technology Category

• Humans and AI: Learning Human Values and Preferences
• Philosophy and Ethics of AI: Morality & Value-based AI
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Aligning Large Language Models to integrate and reflect human values, especially for tasks that demand intricate human oversight, is arduous since it is resource-intensive and time-consuming to depend on human expertise for context-specific guidance. Prior work has utilized predefined sets of rules or principles to steer the behavior of models (Bai et al., 2022; Sun et al., 2023). However, these principles tend to be generic, making it challenging to adapt them to each individual input query or context. In this work, we present Situated-PRInciples (SPRI), a framework requiring minimal or no human effort that is designed to automatically generate guiding principles in real-time for each input query and utilize them to align each response. We evaluate SPRI on three tasks, and show that 1) SPRI can derive principles in a complex domain-specific task that leads to on-par performance as expert-crafted ones; 2) SPRI-generated principles lead to instance-specific rubrics that outperform prior LLM-as-a-judge frameworks; 3) using SPRI to generate synthetic SFT data leads to substantial improvement on truthfulness. We release our code and model generations at https://github.com/honglizhan/SPRI-public.

Problem

• Aligning LLMs with context-specific human values
• Automating real-time principle generation for queries
• Improving LLM performance in domain-specific tasks

Innovation

• Automatically generates real-time guiding principles
• Minimizes human effort in model alignment
• Enhances performance with instance-specific rubrics