TLDR: Deliberate Lab is an open-source, no-code platform designed for real-time social experiments involving both human participants and AI agents. It addresses limitations of existing tools by supporting multi-party interactions at scale, simplifying experiment design, and treating AI as first-class participants. A 12-month deployment showed diverse use cases, from large-scale elections to human-AI negotiation, highlighting its flexibility, real-time monitoring, and streamlined data export capabilities for researchers.
Researchers are increasingly interested in understanding how humans interact, collaborate, and make decisions alongside artificial intelligence. However, the tools available for such studies have often fallen short. Many existing platforms don’t support real-time, multi-party experiments at scale, often require complex custom engineering, and don’t fully integrate AI agents as active participants.
Addressing these challenges, a new open-source platform called Deliberate Lab has been introduced. This platform is designed for large-scale, real-time behavioral experiments, supporting both human participants and agents powered by large language models (LLMs). It aims to lower technical barriers and standardize support for hybrid human-AI experimentation, expanding the methods available for studying collective decision-making and human-centered AI.
Deliberate Lab offers a no-code experiment builder with an intuitive user interface for designing multi-stage experiments. It provides flexible cohort management, allowing researchers to monitor and intervene in live, multi-party interactions as needed. Both human and LLM participants can be included, making it a versatile tool for various research scenarios.
The platform has undergone a 12-month public deployment, involving 88 experimenters and 9,195 participants. Analysis of its usage patterns and workflows, along with in-depth interviews with select experimenters, has provided valuable insights. Researchers from diverse backgrounds, including psychology, economics, and human-computer interaction, have used Deliberate Lab for a wide range of studies, such as large-scale online elections, modeling negotiation dynamics with AI interventions, and developing AI teachers and debate moderators.
Key Features and Design
Deliberate Lab’s system architecture is built on TypeScript, utilizing Google Firebase for its backend, including Firestore for data storage and a Realtime database for tracking participant presence. Cloud Functions handle cohort state updates and agent logic, ensuring real-time interactions.
The platform supports various experiment stages, including chat stages for real-time messaging between participants and agents, survey stages for collecting data, and transfer stages for dynamically sorting participants into different groups. It also includes profile stages for managing participant identities and payout stages for flexible incentive structures.
A crucial aspect of Deliberate Lab is its integration of LLM agents. These agents can act as synthetic participants or as mediators within specific stages. Experimenters can define agents through customizable prompts, allowing for roles like a “Friendly Facilitator” or a “Private Expert Coach” that adapt to conversation history. The platform supports various LLMs compatible with the Chat Completions API format, including Google Gemini and OpenAI GPT, and even custom-hosted models.
Facilitating Experiments in Real Time
Experimenters have access to a dashboard that allows them to preview and manage participants in real time. This includes adding human participants via shared URLs and integrating agent participants with predefined templates and personas. A unique feature is the ability to manually transfer participants between cohorts or remove them during a session, enabling dynamic reconfiguration of groups.
Real-time monitoring tools are essential for maintaining participant engagement in online collaborative experiments. Status indicators show when a participant is inactive, and presence detection displays whether users are connected or active. Facilitators can send attention checks, message participants directly, or trigger alerts for assistance. LLM agents can also be monitored via a debugging panel, allowing for real-time adjustments to their settings and prompts.
Participant Experience
Participants interact with a user-friendly interface that includes a stage navigation menu, main content area, and profile display. They advance through stages, can revisit previous ones, and experience wait screens for synchronized group activities. Features like cohort progress bars visualize peers’ real-time progress, and participants can trigger alerts for help.
Also Read:
- Crafting Human-Like AI: A New Framework for Emotional Cognition in Virtual Agents
- A New Standard for Assessing AI Agent Performance
Impact and Future Directions
The deployment of Deliberate Lab has demonstrated its utility across various use cases. Experimenters praised its flexibility, accessibility, and the ability to run complex, synchronous experiments that would have been otherwise infeasible. The platform’s real-time monitoring, streamlined data export, and scalability were highlighted as significant advantages.
While powerful, the platform acknowledges limitations and opportunities for future work. These include enhancing scalability and facilitation complexity with features like anomaly detection, supporting multi-modality (audio/video), and integrating model development workflows for data annotation and fine-tuning. The researchers emphasize that LLM agents should complement, not replace, human-subject research, and future work may explore longitudinal studies and persistent agent identities.
Deliberate Lab represents a significant step forward in research infrastructure for studying digitally mediated social behavior. By making human-AI collective experimentation more accessible, interdisciplinary, and reproducible, it provides a standard tool for understanding how groups reason, deliberate, and decide in an algorithmic age. You can learn more about this innovative platform by reading the full research paper here: Deliberate Lab Research Paper.
