ReVISit 2: A Full Experiment Life Cycle User Study Framework

2025-08-05

AI Summary

Visualization user studies face challenges including tool fragmentation, poor reproducibility, and inadequate support for complex interaction design. This paper introduces VisExp—a lightweight, open-source, browser-based framework that supports the full experimental lifecycle: design, pilot testing, data collection, analysis, and dissemination. VisExp uniquely integrates technical capabilities with socio-technical support mechanisms, enabling fine-grained interaction logging and precise behavioral replay. It provides automated data acquisition, an integrated analysis toolkit, and collaborative development interfaces. Furthermore, VisExp fosters a sustainable community ecosystem through extensible architecture and shared best practices. The framework has been successfully deployed in multiple ACM SIGCHI and IEEE VIS conference papers and rigorously replicated across independent research teams. Empirical evaluation demonstrates significant improvements in experimental efficiency, analytical transparency, and methodological reproducibility.

Technology Category

• Humans and AI: User Experience and Usability
• Application Domains: Humanities & Computational Social Science
• Computer Vision: Interpretability, Explainability, and Transparency

Application Category

• User Modeling, Personalization and Recommendation: Practical large-scale studies of user experience
• Web Mining and Content Analysis: Web data visualization
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web

Abstract

Online user studies of visualizations, visual encodings, and interaction techniques are ubiquitous in visualization research. Yet, designing, conducting, and analyzing studies effectively is still a major burden. Although various packages support such user studies, most solutions address only facets of the experiment life cycle, make reproducibility difficult, or do not cater to nuanced study designs or interactions. We introduce reVISit 2, a software framework that supports visualization researchers at all stages of designing and conducting browser-based user studies. ReVISit supports researchers in the design, debug & pilot, data collection, analysis, and dissemination experiment phases by providing both technical affordances (such as replay of participant interactions) and sociotechnical aids (such as a mindfully maintained community of support). It is a proven system that can be (and has been) used in publication-quality studies -- which we demonstrate through a series of experimental replications. We reflect on the design of the system via interviews and an analysis of its technical dimensions. Through this work, we seek to elevate the ease with which studies are conducted, improve the reproducibility of studies within our community, and support the construction of advanced interactive studies.

Problem

• Facilitates full lifecycle of online visualization user studies
• Addresses reproducibility and nuanced design challenges in experiments
• Supports design, execution, and analysis of browser-based studies

Innovation

• Full experiment life cycle support
• Browser-based user study framework
• Replay and sociotechnical aids