Co-design of LLM-based preference agents: participation may drive overtrust

2026-07-23

AI Summary

This study addresses the challenges large language models (LLMs) face in accurately simulating human preferences, including difficulties in validation, representational bias, and risks of excluding marginalized groups, compounded by participatory design potentially masking insufficient alignment. Through a three-phase process—comprising background questionnaires, co-design interviews, and validation questionnaires—twelve participants collaboratively developed personalized preference agents in a household energy context. Qualitative analysis assessed human–AI alignment, revealing that while users generally perceived their agents as accurately reflecting their preferences, independent validation showed the agents’ responses were more homogeneous, decisive, and abstract, indicating limited actual alignment. The findings suggest that participatory design may function as an “overtrust engine,” advocating for individual alignment to be understood as a dynamic practice rather than a static state, and highlighting structural bias risks in large-scale deployments.

Abstract

Large language models are increasingly used to simulate human preferences in research and practical applications, raising concerns about validation, misrepresentation, and exclusion. Co-designing agents with the people they represent is a promising way to address these concerns, but participation may also mask the problems it appears to solve. This paper explores that tension through a primarily qualitative study in which 12 participants co-designed personal preference agents in the domain of household energy, via a background survey, co-design interview, and validation survey. Participants engaged readily and mostly came to see their agents as representing them well. Independent validation, however, revealed mixed human-agent alignment, with agent responses markedly more homogeneous, decisive, and abstract than the human sample. I argue that participation and process transparency can act as an "overtrust engine" that promotes trust while concealing systematic misalignment with potential structural consequences at scale. I develop this as a core mechanism in participatory preference agent design, treating individual alignment not as a fixed state but as an enacted process.

Problem

• preference agents
• large language models
• co-design
• overtrust
• human-agent alignment

Innovation

• co-design
• preference agents
• overtrust
• human-agent alignment
• large language models