TLDR: The APEX system introduces human-AI co-embodied intelligence, integrating human operators, agentic AI, and wearable mixed-reality hardware to transform scientific experimentation and manufacturing. It provides real-time guidance, error correction, and automated documentation, significantly improving accuracy, reproducibility, and skill transfer for both experienced and novice users in complex physical tasks like microfabrication.
Scientific research and manufacturing often involve intricate, multi-step procedures that demand continuous human expertise for precise execution and critical decision-making. Despite significant advancements in machine learning and automation, traditional AI models have largely been confined to virtual environments, focusing on data analysis, outcome prediction, or code generation. Real-world experimentation and manufacturing, however, have continued to rely heavily on human supervision and specialized skills. This disparity between digital intelligence and physical execution has created limitations in reproducibility, scalability, and accessibility across various scientific and manufacturing workflows.
To address this crucial gap, researchers have introduced a groundbreaking concept: human-AI co-embodied intelligence. This innovative form of physical AI integrates human researchers or operators, intelligent AI agents, and wearable hardware into a unified system designed for real-world experimentation and smart manufacturing. In this paradigm, humans contribute their unparalleled precision in execution and delicate motor control, while agentic AI provides essential memory, contextual reasoning, adaptive planning, and real-time feedback.
As a prime example of this new intelligence, the Agentic-Physical Experimentation (APEX) system has been developed. APEX couples agentic reasoning with physical execution through mixed-reality (MR) interaction. It’s designed to observe and interpret human actions, align them with dynamic standard operating procedures (SOPs), offer 3D visual guidance, and automatically document and evaluate every step of a process. The system has been successfully implemented in a cleanroom setting for flexible electronics fabrication, demonstrating remarkable capabilities.
How APEX Works
The APEX system is built upon a sophisticated architecture that includes a real-time MR hardware platform, equipped with advanced MR goggles for immersive visual interaction and spatial mapping, and multiple multimodal Large Language Model (LLM)-driven AI agents. These expert-trained agents work collaboratively to enable planning, perception, reasoning, and adaptive feedback. When a human uses APEX in a cleanroom, the system continuously interprets egocentric MR video streams from the fabrication environment, aligning observed scenes with the corresponding SOP steps.
The core of APEX’s reasoning framework consists of four specialized agents:
- Planning Agent: Translates user intent into an executable plan, generating and sequencing SOPs for individual fabrication steps. It can even expand its knowledge by referencing online literature for new procedures.
- Context Agent: Acts as a vision grounding module, linking real-world fabrication data (equipment, materials, user actions) to structured knowledge from the experiment plan.
- Step-tracking Agent: Monitors user progress by aligning current visual information with the SOP steps, dynamically adjusting its memory and prediction intervals to ensure robust step prediction.
- Analysis Agent: Maintains an evolving memory of the fabrication procedure, providing real-time feedback, answering user queries, detecting procedural errors, and offering instant corrections and next-action guidance.
The MR goggles are crucial to this system, providing high-resolution video input, hand and eye tracking data, and a reconstructed 3D map of the environment. This multimodal perception allows the AI agents to understand the physical layout, instrument positions, and user movements, facilitating highly accurate contextual reasoning.
Benefits for Experienced Users
Even experienced users in complex environments like cleanrooms can benefit significantly from APEX. The system enhances microfabrication performance by:
- Real-time Error Detection and Correction: APEX continuously interprets user gestures and system states, providing adaptive, step-specific suggestions. It can detect deviations from SOPs, such as incorrect parameter entries, and issue immediate MR alerts for correction, preventing costly mistakes.
- Autonomous Documentation: The system automatically generates structured experimental summaries, recording parameters, timestamps, environmental snapshots, and actions. This creates a comprehensive, searchable digital log for reproducibility and quality assurance.
- Human-AI Q&A: Users can query the Analysis agent at any time to review previous operations, parameter deviations, and outcomes through natural-language interactions, grounded in the structured experimental memory.
This bidirectional interaction transforms fabrication from an isolated manual endeavor into an interpretable, collaborative process, amplifying precision, consistency, and reproducibility.
Accelerating Skill Transfer for Beginners
One of APEX’s most impactful features is its ability to accelerate skill transfer to inexperienced researchers. Traditionally, mastering microfabrication skills takes months to years of mentorship. APEX drastically shortens this learning curve by:
- Expert Behavior Mirroring: The system records and organizes contextual data from expert fabrications. When assisting novices, APEX retrieves this structured memory and corresponding SOPs to provide step-specific, real-time guidance that mirrors expert behavior.
- Interactive Training: The MR interface provides real-time, step-specific visual cues synchronized with the user’s movements and system state, enabling clear and intuitive interaction. It detects action deviations and issues corrections, preventing errors.
In demonstrations, inexperienced researchers guided by APEX rapidly approached the performance curve of experienced researchers, achieving expert-level proficiency within a single session. This capability transforms skill acquisition into a structured, interactive, and interpretable training experience.
Also Read:
- Advancing Robot Dexterity: A Collaborative AI and VR Framework for Enhanced Manipulation
- Real-DRL: Bridging the Gap for Safe AI in Physical Systems
A New Era of Scientific Manufacturing
The APEX system represents a new paradigm of Physical AI, where agentic reasoning is directly coupled with embodied human execution. By integrating real-time perception, multi-agent reasoning, and adaptive MR interaction, APEX transforms microfabrication into a transparent, interpretable, and scalable process. It functions as an active collaborator, interpreting operator gestures, providing predictive guidance, and correcting procedural deviations in real time.
Looking ahead, the APEX system has the potential for further improvements in both hardware and software, such as lighter MR goggles, on-device LLM deployment for enhanced security, and personalized memory systems for individual users. Its framework can also be extended to other laboratory domains like materials synthesis, biomedical experimentation, or robotics assembly, where interpretability, adaptability, and reproducibility are paramount. Ultimately, APEX lays the groundwork for a new generation of self-evolving, collaborative scientific manufacturing systems. You can read the full research paper here: Human-AI Co-Embodied Intelligence for Scientific Experimentation and Manufacturing.
