Values in the Wild: Discovering and Analyzing Values in Real-World Language Model Interactions

2025-04-21

AI Summary

Prior research lacks empirical characterization of AI systems’ implicit value orientations in authentic user interactions. Method: This study introduces a privacy-preserving, bottom-up value extraction framework, analyzing hundreds of thousands of real-world dialogue logs to construct an interpretable taxonomy of 3,307 normative values—representing the first large-scale, empirically grounded value classification for LLMs (exemplified by Claude 3/3.5). Leveraging response-content-driven value identification, context-aware attribution, and unsupervised annotation, the method captures value expression as inherently context-dependent. Contribution/Results: Findings reveal strong contextual specificity (e.g., “historical accuracy” emerges exclusively in contentious event queries), challenging static value assumptions. The study maps the AI value distribution landscape, confirming a robust pro-social bias, resistance to moral nihilism, and salient context-sensitive principles—including transparency, harm prevention, and human agency.

Technology Category

• Humans and AI: Learning Human Values and Preferences
• Philosophy and Ethics of AI: Morality & Value-based AI
• Natural Language Processing: Ethics — Bias, Fairness, Transparency & Privacy

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Accountability, Transparency, and Ethics for personalization
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems

Abstract

AI assistants can impart value judgments that shape people's decisions and worldviews, yet little is known empirically about what values these systems rely on in practice. To address this, we develop a bottom-up, privacy-preserving method to extract the values (normative considerations stated or demonstrated in model responses) that Claude 3 and 3.5 models exhibit in hundreds of thousands of real-world interactions. We empirically discover and taxonomize 3,307 AI values and study how they vary by context. We find that Claude expresses many practical and epistemic values, and typically supports prosocial human values while resisting values like"moral nihilism". While some values appear consistently across contexts (e.g."transparency"), many are more specialized and context-dependent, reflecting the diversity of human interlocutors and their varied contexts. For example,"harm prevention"emerges when Claude resists users,"historical accuracy"when responding to queries about controversial events,"healthy boundaries"when asked for relationship advice, and"human agency"in technology ethics discussions. By providing the first large-scale empirical mapping of AI values in deployment, our work creates a foundation for more grounded evaluation and design of values in AI systems.

Problem

• Identify values AI models exhibit in real-world interactions
• Analyze how AI values vary by context and topic
• Establish foundation for evaluating and designing AI values

Innovation

• Privacy-preserving method for value extraction
• Taxonomize 3,307 AI values empirically
• Context-dependent value mapping in AI