TLDR: The paper introduces “Prompt-and-Check,” a method using open-source Large Language Models (LLMs) like LLaMA 3 8B to evaluate communication protocol compliance in simulation-based training. It efficiently assesses transcribed verbal exchanges against checklists using context-rich prompts and a two-stage temporal and semantic context selection. The approach, deployable on consumer-grade GPUs, achieves high accuracy (LLaMA 3 8B at 93.6%) without task-specific training, demonstrating LLMs’ utility for automated assessment and feedback in safety-critical domains.
In high-stakes environments like maritime operations, healthcare, and aviation, clear and precise communication is not just good practice—it’s critical for safety. Ensuring that professionals adhere to established communication protocols during emergencies or routine procedures is vital for operational competence. Traditionally, evaluating this adherence has been a labor-intensive process, often relying on experts manually reviewing transcripts, logs, or recordings of verbal exchanges.
A new research paper, “Prompt-and-Check: Using Large Language Models to Evaluate Communication Protocol Compliance in Simulation-Based Training,” introduces an innovative and efficient approach to automate this crucial assessment. Authored by Vishakha Lall and Yisi Liu from the Centre of Excellence in Maritime Safety at Singapore Polytechnic, this study explores how open-source Large Language Models (LLMs) can be leveraged to evaluate communication protocol compliance in simulation-based training environments.
The core of their method, dubbed “Prompt-and-Check,” involves feeding context-rich prompts to LLMs. These prompts are designed to help the LLM determine whether each item on a communication protocol checklist has been fulfilled, based solely on transcribed verbal interactions. What makes this approach particularly noteworthy is its efficiency: it’s lightweight and can run effectively on consumer-grade GPUs, such as an RTX 4070, making it highly deployable.
The researchers conducted a case study within the maritime domain. They had participants perform identical simulation tasks, exposing them to high-risk events like potential vessel collisions, engine failures, and severe storms. The verbal communications during these simulations were recorded and transcribed. For each checklist item, a prompt containing relevant transcript excerpts was fed into models like LLaMA 2 7B, LLaMA 3 8B, and Mistral 7B. The models then output a judgment on compliance.
A key innovation in the “Prompt-and-Check” methodology is its sophisticated context selection. Instead of feeding the entire, potentially noisy, transcript to the LLM, the system first extracts a temporal window of communication relevant to a specific event. This is then further refined using semantic similarity filtering, ensuring that only the most pertinent utterances are presented to the LLM. This dual-filtering approach significantly improves accuracy and computational efficiency, as demonstrated in their ablation study.
The LLMs’ outputs were assessed against expert-annotated ground truth using metrics like classification accuracy and justification alignment scores. The findings were compelling: the prompting approach enabled effective, context-aware reasoning without requiring any task-specific training (known as zero-shot inference). Among the models tested, LLaMA 3 8B consistently outperformed the others, achieving the highest overall compliance accuracy of 93.6% and a strong justification alignment score of 1.8 (on a 0-2 scale).
This study highlights the practical utility of LLMs in enhancing various aspects of training, including debriefing, performance feedback, and automated assessment. The “Prompt-and-Check” method offers a generalizable framework that could be applied across other safety-critical domains like aviation, emergency response, and industrial safety, where adherence to verbal protocols is paramount. The paper can be found here.
Also Read:
- Beyond Generic Responses: How AI Tutors Can Learn to Guide Students More Effectively
- Adaptive Prompting Strategies Boost Biomedical Named Entity Recognition
The researchers envision future work exploring the generalization of this method across different types of protocols, incorporating multimodal inputs (such as video or sensor data), and extending its use for real-time decision support or debriefing tools. This research marks a significant step towards automating and improving the evaluation of human communication in critical operational settings, ultimately contributing to enhanced safety and competence.
