Deliberate Lab: A Platform for Real-Time Human-AI Social Experiments

2025-10-14

AI Summary

Human–AI collaboration research lacks scalable, real-time, multi-party experimental platforms. Method: This paper introduces the first standardized social experimentation framework treating AI agents as first-class participants. Built upon large language models (LLMs), it implements programmable AI agents integrated within a real-time, multi-user web interface and an open-source, extensible architecture—substantially lowering barriers to experiment development and replication. Contribution/Results: The framework formally establishes AI agents as co-equal experimental subjects and provides a unified interface supporting hybrid human–AI decision-making paradigms. Deployed publicly for 12 months, it enabled 88 researchers to conduct experiments involving 9,195 human participants. Empirical evaluation confirms its advantages in scalability, usability, and scientific validity.

Technology Category

• Humans and AI: Intelligent User Interfaces
• Multiagent Systems: Adversarial Agents
• Cognitive Modeling & Cognitive Systems: Agent Architectures

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems

Abstract

Social and behavioral scientists increasingly aim to study how humans interact, collaborate, and make decisions alongside artificial intelligence. However, the experimental infrastructure for such work remains underdeveloped: (1) few platforms support real-time, multi-party studies at scale; (2) most deployments require bespoke engineering, limiting replicability and accessibility, and (3) existing tools do not treat AI agents as first-class participants. We present Deliberate Lab, an open-source platform for large-scale, real-time behavioral experiments that supports both human participants and large language model (LLM)-based agents. We report on a 12-month public deployment of the platform (N=88 experimenters, N=9195 experiment participants), analyzing usage patterns and workflows. Case studies and usage scenarios are aggregated from platform users, complemented by in-depth interviews with select experimenters. By lowering technical barriers and standardizing support for hybrid human-AI experimentation, Deliberate Lab expands the methodological repertoire for studying collective decision-making and human-centered AI.

Problem

• Developing infrastructure for real-time human-AI social experiments
• Addressing limited replicability in hybrid human-AI behavioral studies
• Treating AI agents as participants in collective decision-making research

Innovation

• Open-source platform for real-time human-AI experiments
• Supports both human participants and LLM-based agents
• Standardizes hybrid experimentation to lower technical barriers