TLDR: LabOS is a pioneering AI co-scientist system that integrates advanced AI reasoning with physical laboratory experimentation through extended reality (XR) smart glasses. It enables AI to perceive, understand, and interact in real-time lab settings, offering guidance, detecting errors, and automating documentation. LabOS has shown success in diverse biomedical applications, including cancer immunotherapy target discovery, mechanistic research, and stem cell engineering, aiming to accelerate scientific breakthroughs and improve reproducibility by fostering human-AI collaboration.
Imagine a future where artificial intelligence doesn’t just crunch numbers or design experiments on a computer, but actively participates in the physical laboratory, seeing what scientists see, understanding the context of experiments, and offering real-time assistance. This future is becoming a reality with LabOS, a groundbreaking AI co-scientist system that unites computational reasoning with hands-on physical experimentation.
LabOS addresses a critical gap in modern science. While AI has revolutionized computational tasks like simulation and prediction, the physical lab often remains a bottleneck. Traditional AI agents largely operate in the digital realm, and robotic automation, though powerful, can be rigid and costly to adapt. LabOS steps in to bridge this divide, creating a unified human-AI collaborative intelligence platform that makes laboratories both AI-perceivable and AI-operable.
At its core, LabOS features a sophisticated multi-agent AI architecture. This includes a Manager Agent for planning, a Developer Agent for executing complex analyses, a Critic Agent for evaluating results, and a Tool-Creation Agent that continuously expands the system’s capabilities by generating new tools from scientific literature and data. This self-improving design allows LabOS to learn and evolve, tackling novel research tasks with increasing efficiency.
The system connects AI reasoning directly to the laboratory through Extended Reality (XR) smart glasses. Researchers wearing these glasses receive adaptive, context-aware guidance from the AI. This includes step-by-step instructions, real-time error detection and correction cues, and even gesture or voice interactions for sterile workflows. To enable the AI to “see” in the lab, a specialized Vision-Language Model (VLM), called LabOS VLM, was trained on extensive lab video data. This VLM can interpret visual input from the XR glasses, understand lab scenes, monitor actions, detect deviations from protocols, and verify results, effectively allowing the AI to co-pilot experiments alongside human scientists.
LabOS also supports advanced 3D and 4D spatial modeling of laboratory workflows. These “digital twins” capture the spatial and temporal relationships between instruments, samples, and human actions, enabling simulations, “what-if” analyses, and simulation-based training, all contributing to safer, more reproducible, and transferable laboratory automation.
The impact of LabOS has been demonstrated across several biomedical research studies. In cancer immunology, LabOS successfully identified CEACAM6 as a potential target to boost natural killer (NK) cell killing of tumors, a finding later validated in wet-lab experiments. For mechanistic research, LabOS proposed ITSN1 as a key gene regulating cell-cell fusion, which was also confirmed experimentally. In stem cell engineering, researchers used LabOS with XR glasses to guide complex gene-editing experiments, where the AI monitored workflows, provided guidance, and flagged operational deviations. LabOS can even record expert practices and digitize key parameters, acting as an AI tutor to train novice researchers to achieve expert-level performance.
Also Read:
- Autonomous Agents Chart a New Course for Scientific Discovery
- Deliberate Lab: A New Platform for Human-AI Social Experimentation
By giving AI the ability to think with us and work alongside human scientists – seeing what we see, checking what we do, and learning from every run – LabOS transforms the lab into a dynamic, collaborative environment. It moves us closer to autonomous, self-improving discovery, where human intuition and machine rigor co-evolve to accelerate scientific breakthroughs. To learn more about this innovative system, you can read the full research paper here.
