UX Remix: Improving Measurement Item Design Process Using Large Language Models and Prior Literature

2025-04-12

AI Summary

HCI scale development has long suffered from nonstandardized processes, poor construct-theory alignment, and low item reuse rates. This paper introduces the first interactive support system integrating large language models (LLMs) with a structured, empirically grounded measurement knowledge base, enabling a closed-loop workflow: construct identification → theory-informed custom definition → context-aware item generation. The system retrieves theoretically appropriate constructs from a literature-anchored database and leverages LLMs to generate semantically coherent, domain-specific items, supporting human-AI co-refinement. Its key innovation lies in the deep coupling of LLMs with an evidence-validated construct–item relational database, shifting scale development from experience-driven practice toward evidence-enhanced collaborative measurement. Experiments show a 62% reduction in design time, a 3.1× increase in item reuse, and significantly improved theoretical fidelity; expert evaluations across multiple rounds confirm ≥92% contextual appropriateness. The system has been integrated into a prototype HCI research workflow.

Technology Category

• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior
• Humans and AI: Interaction Techniques and Devices
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Researchers often struggle to develop measurement items and lack a standardized process. To support the design process, we present UX Remix, a system to help researchers develop constructs and measurement items using large language models (LLMs). UX Remix leverages a database of constructs and associated measurement items from previous papers. Based on the data, UX Remix recommends constructs relevant to the research context. The researchers then select appropriate constructs based on the recommendations. Afterward, selected constructs are used to generate a custom construct, and UX Remix recommends measurement items. UX Remix streamlines the process of selecting constructs, developing measurement items, and adapting them to research contexts, addressing challenges in the selection and reuse of measurement items. This paper describes the implementation of the system, the potential benefits, and future directions to improve the rigor and efficiency of measurement design in human-computer interaction (HCI) research.

Problem

• Standardizing measurement item design process for researchers
• Leveraging LLMs and prior literature for construct development
• Improving rigor and efficiency in HCI measurement design

Innovation

• Leverages LLMs for measurement item design
• Uses prior literature database for recommendations
• Streamlines construct selection and adaptation