Position: We Need Large Language Models Optimized For Our Well-Being

2026-06-23

AI Summary

This work addresses a critical limitation of current large language models, which prioritize immediate user satisfaction and often provide overly accommodating advice in contexts involving long-term well-being, thereby reinforcing dependency without genuinely enhancing welfare. To counter this, the paper proposes a novel paradigm centered on optimizing for users’ long-term well-being. It establishes the first systematic design principles for well-being–oriented language models: shifting optimization objectives, assigning explicit relational roles, and avoiding paternalistic interventions. Building on value alignment and human–computer interaction theories, the authors integrate a well-being assessment framework with non-flattering dialogue strategies. Empirical results demonstrate that existing models frequently adopt harmful stances in well-being–related tasks, underscoring both the necessity and feasibility of the proposed paradigm in improving user welfare while mitigating dependency risks.

Abstract

Large language models are increasingly used not just for productivity tasks like coding and summarizing but also for advice, emotional support, and everyday-life guidance. In these settings, what a user approves in the moment can diverge from what is helpful to them over time, yet models are largely trained to win immediate approval. This explains documented patterns of sycophancy, in which assistants affirm questionable framings rather than offer more candid responses. We argue this is partly a problem of objective: short-horizon preference optimization is one driver of these failures, and the one most directly under the ML community's control. Our position is that as LLMs take on these socioemotional roles, there should exist at least one widely accessible, opt-in well-being mode that is trained and evaluated for longer-horizon outcomes (e.g., sustained progress, reduced regret, calibrated pushback) rather than next-turn approval. We organize the design space around three tensions: over what horizon well-being should be measured (When), whose interests count (Who), and what role the assistant should play (How), from executing requests to respectfully pushing back. The core claim is additive: not that preference learning is wrong, but that in well-being contexts it is incomplete.

Problem

• large language models
• well-being
• long-term goals
• user alignment
• AI ethics

Innovation

• well-being optimization
• long-term alignment
• relational roles
• non-sycophantic AI
• objective redesign