Facilitating Longitudinal Interaction Studies of AI Systems

2025-08-13

AI Summary

Current AI systems research predominantly relies on one-off evaluations, failing to capture the dynamic, longitudinal interaction between users and systems—including learning, adaptation, and repurposing—thereby creating methodological bottlenecks in long-term deployment, evaluation design, and data collection. To address this, this project organized a UIST workshop focused on practical challenges of longitudinal interaction research, integrating keynote presentations, small-group discussions, and hands-on sessions. It systematically proposed deployment strategies, evaluation protocols, and data collection frameworks tailored for sustained studies, and developed reusable tool prototypes. Key contributions include: (1) a structured longitudinal research framework; (2) an open-source toolchain supporting long-term human-AI interaction studies; and (3) an emerging cross-institutional academic community. The project advances longitudinal interaction research from a niche practice to a mainstream methodology in human-computer interaction, significantly enhancing researchers’ capacity to design and conduct sustained, real-world human-AI collaboration studies.

Technology Category

• Humans and AI: Interaction Techniques and Devices
• Cognitive Modeling & Cognitive Systems: Social Cognition And Interaction
• Multiagent Systems: Teamwork

Application Category

• User Modeling, Personalization and Recommendation: Studies of user behavior, including longitudinal effects of personalized systems
• Economics, Online Markets and Human Computation: Research challenges in human and human-AI computation
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web

Abstract

UIST researchers develop tools to address user challenges. However, user interactions with AI evolve over time through learning, adaptation, and repurposing, making one time evaluations insufficient. Capturing these dynamics requires longer-term studies, but challenges in deployment, evaluation design, and data collection have made such longitudinal research difficult to implement. Our workshop aims to tackle these challenges and prepare researchers with practical strategies for longitudinal studies. The workshop includes a keynote, panel discussions, and interactive breakout groups for discussion and hands-on protocol design and tool prototyping sessions. We seek to foster a community around longitudinal system research and promote it as a more embraced method for designing, building, and evaluating UIST tools.

Problem

• Studying evolving user-AI interactions over time
• Addressing challenges in longitudinal research implementation
• Developing practical strategies for long-term system studies

Innovation

• Develop tools for longitudinal AI interaction studies
• Workshop with protocol design and prototyping sessions
• Foster community for longitudinal system research