TLDR: A new configurable research platform allows HCI researchers to conduct controlled experiments on human-LLM agent collaboration. It bridges the gap between human-human and multi-agent AI studies by integrating LLM agents into classic collaboration tasks, enabling systematic investigation of how interaction controls affect human-AI teamwork and user perceptions.
In the rapidly evolving landscape of artificial intelligence, the way humans and AI systems work together is becoming increasingly important. Traditionally, AI has been seen as a tool, assisting humans with tasks. However, recent advancements in large language models (LLMs) are opening doors for AI to become more like true collaborators, engaging in natural communication and exhibiting human-like social and cognitive behaviors.
This shift raises crucial questions for researchers in Human-Computer Interaction (HCI) and Computer-Supported Cooperative Work (CSCW): Do the established principles of human-human collaboration still apply when humans team up with LLM agents? To systematically explore these questions, a new research platform has been introduced. This platform is designed to be open and highly configurable, allowing HCI researchers to conduct controlled experiments on human-LLM agent collaboration.
The platform’s core architecture is built around four key modules. First, a researcher interface allows scientists to set up experiments, monitor live sessions, and analyze results. Second, a participant interface enables individuals to interact with both other humans and LLM agents within the experiment. Third, an Agent Context Protocol (ACP) standardizes how LLM agents are integrated, ensuring they operate under consistent rules. Finally, an experiment controller manages the overall state of the experiment, updates actions, and logs all interactions into a database.
One of the significant challenges in this field has been the lack of a flexible research platform that can adapt classic CSCW experiments for human-agent collaboration. Re-implementing these experiments from scratch for AI agents is time-consuming and often lacks reusability. This new platform addresses this by offering a modular design that can seamlessly adapt existing experimental paradigms, such as the “Shape Factory” experiment, “DayTrader,” and “Essay Ranking.” Researchers can manipulate theory-grounded interaction controls like communication methods, awareness dashboards, and social framing to study their impact on collaborative dynamics.
The platform’s utility was demonstrated through a two-part evaluation. In the first part, two case studies re-implemented the classic “Shape Factory” experiment, transforming a human-human collaboration task into a human-agent collaboration scenario with one human and five LLM agents. These studies manipulated communication levels and awareness levels, showing that the platform could capture significant differences in participant behaviors and outcomes. For instance, varying communication modality led to distinct patterns in negotiation frequency and team performance, aligning with established findings in collaborative work.
The second part of the evaluation involved a participatory cognitive walkthrough with five experienced HCI researchers. This helped refine the researcher interface, making it easier for scientists to set up and analyze experiments. Feedback led to improvements in onboarding, parameter clarity, terminology, and information architecture, ensuring the platform is user-friendly for its target audience.
This research is a methodological contribution, bridging the gap between traditional human-human collaboration studies and the rapidly advancing field of LLM agents. It provides a controlled and reproducible environment for understanding how humans and AI can effectively work together as partners, rather than just tools. By standardizing agent integration through the Agent Context Protocol, the platform ensures that agents perceive and act within the experiment under the same constraints as human participants, crucial for valid comparative studies.
The platform also helps translate high-level design guidelines for human-AI interaction into specific, testable hypotheses. For example, researchers can explore different levels of agent explainability or initiative and observe their impact on collaborative dynamics. This approach allows for a systematic re-examination of foundational CSCW theories in the context of human-agent teams, investigating which principles remain valid, which need revision, and what new phenomena emerge.
With its customizability and generalizability, the platform is poised to benefit interdisciplinary researchers. Agent researchers can use it as a human-centered benchmark to gain deeper insights into the social and cognitive alignment of agent behaviors. Social scientists can program agents to adopt specific strategies to study human team reactions, offering a level of control impossible with human confederates. HCI researchers can create domain-specific “scenario packs” to model real-world collaborative workflows while maintaining experimental control. The commitment to open-science practices, including open-source code and configuration profiles, encourages community co-development and fosters comparable, extensible studies of human-agent collaboration.
Also Read:
- Beyond the Words: How AI Interface Design Shapes a Writer’s Control and Authorship
- Navigating the Future: A Deep Dive into Generalizability for LLM-Based Agents
While the fidelity of LLM agents and the generalizability of findings from abstract lab tasks to real-world contexts remain areas for future work, this platform establishes a robust foundation for a systematic, evidence-based science of human-agent collaboration. You can read the full paper here.
