TLDR: A research paper introduces SapienAI, a multi-modal AI system using speech analysis, computer vision, and sentiment detection to assess and improve public speaking skills in engineering students. It uniquely evaluates “expressive coherence” – the alignment of verbal and non-verbal cues – and found Gemini 1.5 Pro to be the best-performing LLM for this task, showing moderate agreement with human experts. This system offers personalized, scalable feedback, addressing limitations of traditional training.
Effective communication is a cornerstone of success for engineers, enabling them to bridge the gap between complex technical innovations and diverse stakeholders. However, traditional university curricula often fall short in providing comprehensive and personalized public speaking training, leaving many engineering students without the nuanced skills needed for impactful presentations.
Recognizing this persistent challenge, researchers at Plaksha University have developed an innovative AI-driven assessment model called SapienAI. This groundbreaking system aims to enhance public speaking skills in engineering students by offering sustained and personalized feedback, addressing the limitations of time-intensive human evaluations.
A Multi-Modal Approach to Public Speaking Assessment
SapienAI stands out by integrating multiple AI technologies to provide a holistic evaluation of public speaking. Unlike previous systems that might assess verbal and non-verbal aspects separately, SapienAI combines speech analysis, computer vision, and sentiment detection into a unified multi-modal AI system. This allows it to evaluate three key areas:
- Verbal Communication: Analyzing elements like pitch, loudness, pacing, and intonation.
- Non-Verbal Communication: Assessing facial expressions, gestures, and posture.
- Expressive Coherence: A novel integration that ensures alignment between a speaker’s verbal articulation and their body language and emotional cues. This is a crucial aspect often missed by other assessment tools.
The system works by collecting data from student presentations in a controlled setting, using high-definition video and microphones. This data is then meticulously analyzed: speech is transcribed and segmented, vocal features (like pitch and speech rate) are extracted using libraries like Librosa, and non-verbal indicators (facial expressions, body pose, hand movements) are identified using tools such as DeepFace and Mediapipe. All this rich, multi-modal data is then compiled and fed into Large Language Models (LLMs) for evaluation.
Benchmarking and Performance
To rigorously test SapienAI, the researchers benchmarked four state-of-the-art LLMs—Gemini 1.5 Pro, Gemini 2.0 Flash, GPT-4o Mini, and O1 Mini—against expert human raters. The evaluation used a customized 12-criterion rubric, which included traditional public speaking metrics alongside two new criteria introduced by this research: ‘dynamic emphasis’ (how verbal and non-verbal signals jointly highlight key points) and ’emotional resonance’ (the convergence of emotional content through both words and physical demeanor).
The results were promising: Gemini 1.5 Pro emerged as the best-performing model, achieving a ‘moderate agreement’ (Cohen’s Kappa of 0.41) with human annotators across all rubrics. Notably, Gemini 1.5 Pro particularly excelled on rubrics that required the integration of multiple modalities, such as vocal expression and the newly introduced expressive coherence criteria. In some instances, the AI model even demonstrated superior judgment compared to human graders, accurately identifying subtle non-verbal cues that human evaluators might have overlooked.
Also Read:
- AI Breakthrough: Automating the Extraction of Criminal Facts from Court Opinions
- Beyond Universal Safety: Stress-Testing LLMs with Custom Behavioral Policies
Broader Impact and Future Directions
The implications of SapienAI extend beyond engineering education. This multi-modal AI framework has the potential to transform scalable communication training across various disciplines by automating feedback and reducing reliance on one-on-one coaching. The concept of “expressive coherence” also paves the way for more holistic AI-driven evaluations in fields like remote collaboration and mental health assessments, capturing how speech, gestures, and emotional signals align.
While the initial findings are highly encouraging, the researchers acknowledge limitations, such as the current feature set not fully capturing the diversity in public speaking styles across different genders and cultural backgrounds. Future work will explore integrating physiological data (like heart rate and skin temperature) for deeper insights into a speaker’s emotional states and addressing challenges like LLM hallucinations through improved prompt engineering. You can read the full research paper here.
