TLDR: This research introduces an interactive system that allows non-programmers to teach complex collaborative physical activities to AI. By using narrated demonstrations (physical actions paired with natural language), the system synthesizes editable programs. Users can then inspect and correct the AI’s understanding through visual decision flow diagrams and narrated simulations, leading to significant improvements in the AI’s ability to model human intent for tasks like soccer tactics.
Teaching artificial intelligence systems to understand and perform complex physical tasks has long been a goal in human-computer interaction. While significant strides have been made in individual skills, collaborative activities, which require inferring teammates’ intentions and adapting to dynamic situations, present a much greater challenge. A new research paper, Interactive Program Synthesis for Modeling Collaborative Physical Activities from Narrated Demonstrations, introduces an innovative system designed to tackle this very problem, enabling non-technical users to teach AI collaborative tasks without writing a single line of code.
The paper, authored by Edward Kim, Daniel He, Jorge Chao, Wiktor Rajca, Mohammed Amin, Nishant Malpani, Ruta Desai, Antti Oulasvirta, Bjoern Hartmann, and Sanjit Seshia, frames collaborative task learning as a program synthesis problem. This means the system represents learned behaviors as editable programs, making them interpretable and correctable by users. The core of their approach lies in using “narrated demonstrations” – a unified method where users perform physical actions while simultaneously explaining their rationale in natural language. This dual input helps the system understand both the physical nuances and the underlying intent.
How the System Works
The system operates through an interactive loop. First, a user, immersed in a mixed reality (MR) environment via a headset, demonstrates a collaborative task (like a soccer tactic) while narrating their actions. The MR headset records this data, including video, timestamped actions, and transcribed narration. This multimodal data is then processed by a Large Vision Language Model (LVLM), which acts as the program synthesis tool. The LVLM, guided by a library of domain-specific APIs (Application Programming Interfaces) that define possible actions and physical constraints, generates an initial program. This program models the user’s behavior as a hierarchical, interrupt-driven finite state machine, which essentially maps out decision flows and actions based on environmental conditions.
A unique feature of this system is how it communicates its learning back to the user. It does so through two primary methods: decision flow diagrams and narrated program executions. The decision flow diagrams visually represent the underlying code structure, allowing users to inspect the logic without seeing raw code. Nodes and edges in the diagram include descriptions that explain the rationale. For a more dynamic understanding, the system also runs a Unity MR simulation where an avatar performs the learned behavior. Crucially, this avatar narrates its actions and reasoning in real-time using a “Speak()” API embedded in the program, making the AI’s intent transparent to the user.
User Feedback and Refinement
The system is designed for iterative refinement. Users can provide feedback in two ways: by annotating the decision flow diagram to suggest structural changes, or by pausing the program execution in the MR simulation to point, click, and verbally explain corrections directly in the environment. This feedback, combined with the original demonstrations, is fed back to the LVLM, which then repairs and amends the program.
Insights from the User Study
A study involving 20 non-programmer participants, all with competitive soccer experience, evaluated the system’s effectiveness. Participants taught multiplayer soccer tactics in a mixed reality environment. The results were encouraging: 95% found it easy to teach via narrated demonstrations, and 90% found it easy to correct the programs. While initial program alignment with user intent varied, iterative feedback proved highly effective. Program completeness significantly improved after even a single round of feedback, with participants successfully steering the AI towards their intended high-level logic.
However, the study also highlighted areas for improvement. Some participants found the pre-scripted behaviors of environment agents unrealistic, and there was a desire for real-time feedback from the system during teaching. Limitations in the API library sometimes prevented users from implementing desired corrections, and the system’s inability to explain program deadlocks made debugging challenging. Future work aims to address these by incorporating a “Theory of Mind” framework to actively elicit tacit knowledge, expanding API expressiveness, and improving deadlock explanations.
Also Read:
- Interactive Refinement: A New Strategy to Combat Over-fitting in Constraint Acquisition
- AutoPlay: Automating Task Generation for Interactive AI Agents
Conclusion
This research demonstrates a feasible and intuitive approach to teaching complex collaborative physical tasks to AI. By unifying narrated demonstrations for both teaching and system communication, it empowers non-programmers to create and refine executable models of human collaboration. The findings pave the way for future interactive systems that can more effectively acquire and model human knowledge in dynamic, collaborative scenarios.
