🤖 AI Summary
This study addresses a critical gap in current conversational AI systems, which, despite emphasizing goal alignment, lack institutional safeguards centered on the user’s best interests—particularly in high-stakes domains such as mental health and financial decision-making—thereby hindering robust trust and accountability. To bridge this gap, the paper introduces, for the first time, the legal principle of fiduciary duty into the design of conversational AI, proposing a novel normative framework termed “fiduciary design.” Integrating human-computer interaction, AI ethics, and legal analysis, this approach establishes a dialogue agent architecture grounded in fiduciary obligations. By unifying technical trust mechanisms with legal accountability frameworks, the proposed paradigm significantly enhances user trust and strengthens AI accountability in theory, offering an innovative governance pathway for deploying trustworthy AI in high-risk contexts.
📝 Abstract
Conversational agents are increasingly integrated into the most private and intimate aspects of users' lives, from discussions of mental health to financial decisions. As a result, these systems have access to reams of sensitive user data. Much of the literature on AI systems has focused on aligning users' goals with the agents that act on their behalf. While this work is vitally important, it may overlook the need to establish a new normative baseline. Conversational AI agents, designed to feel and interact anthropomorphically with human users, must be held to a standard of care commensurate with their capabilities and access. When a client hires a personal lawyer, undergoes surgery, or receives advice from an investment manager, the expert they consult often has a fiduciary duty to act in their client's best interests. This provocation argues that conversational agents should be held to a similar standard and introduces fiduciary design as a guiding principle. In this respect, conversational AI trust and accountability could be unified into a single design and legal paradigm.