TLDR: Researchers have developed a new AI model for Spoken Language Assessment (SLA) that evaluates a learner’s oral proficiency at the session level, rather than short audio segments. This multimodal foundation model uses multi-target learning and a Whisper-based speech prior to jointly predict overall and part-specific scores in a single pass. It outperforms previous state-of-the-art systems on the Speak & Improve benchmark, offering a more accurate, compact, and deployable solution for language learning applications.
Assessing a learner’s oral proficiency from spontaneous speech, known as Spoken Language Assessment (SLA), is a vital component of Computer Assisted Language Learning (CALL). As the number of English as a Second Language (L2) speakers grows, the demand for reliable and accurate SLA tools intensifies.
Traditional methods for SLA often face significant challenges. Many rely on a series of interconnected steps, which can lead to errors accumulating and propagating through the system. Other end-to-end models typically analyze only short audio segments, potentially missing crucial context and coherence from longer conversations or entire speaking sessions. This creates a disconnect, as human evaluators typically assess a speaker’s proficiency across an entire session, integrating evidence from various parts of a conversation.
A new research paper, titled “SESSION-LEVEL SPOKEN LANGUAGE ASSESSMENT WITH A MULTIMODAL FOUNDATION MODEL VIA MULTI-TARGET LEARNING,” introduces an innovative approach to overcome these limitations. Authored by Hong-Yun Lin, Jhen-Ke Lin, Chung-Chun Wang, Hao-Chien Lu, and Berlin Chen, this work proposes a novel multimodal foundation model designed for session-level evaluation in a single pass. You can read the full paper here.
A Unified Approach to Assessment
The core of this new system lies in its ability to process an entire response session from an L2 speaker coherently. Unlike previous systems that might break down a session into smaller parts and then combine scores, this model evaluates everything at once. It employs a technique called multi-target learning, which allows it to jointly learn both holistic (overall) and trait-level (specific aspects like delivery, language use, and content) objectives of SLA.
A key innovation is the integration of a frozen, Whisper ASR model-based speech prior, referred to as the Acoustic Proficiency Prior (APP). This component provides acoustic-aware calibration, helping the model to better understand non-lexical cues such as fluency, hesitations, and prosody, which are crucial for assessing spoken proficiency but are often lost in text-only or ASR-based pipelines.
How the Model Works
The model uses a Phi-4 Multimodal backbone, a type of generative AI model known for its ability to handle long contexts and integrate speech data. This backbone is fed a dialogue-style sequence that interleaves text-based instructions with the learner’s audio responses. A parallel Whisper branch generates the APP token, which is then prepended to the multimodal sequence. This allows the model to perform session-level reasoning over a unified sequence that includes these vital acoustic cues.
The system is designed to predict a five-dimensional score vector, including individual scores for different parts of a speaking test (e.g., P1, P3, P4, P5) and an overall proficiency score. This multi-target regression approach aligns more closely with how human raters assess performance.
Experimental Validation and Results
The researchers tested their approach on the Speak & Improve (S&I) 2025 benchmark, a realistic dataset for SLA that includes multi-part, open-ended speaking tasks. The results were compelling: the proposed unified, session-level model (Phi-4-MTL-APP) achieved state-of-the-art performance, outperforming previous competitive ensemble and cascaded systems, including the top leaderboard entry in the S&I Challenge 2025, Perezoso.
The model demonstrated robust cross-part generalization and showed clear gains, particularly on multi-response parts (like P1 and P5) and long-audio parts (P3 and P4), where discourse-level reasoning and delivery-sensitive cues are most informative. The integration of the Whisper-derived APP consistently improved accuracy, stabilizing delivery-sensitive cues and enhancing calibration.
Also Read:
- Making AI Conversations Sound More Human: The THINK-VERBALIZE-SPEAK Approach
- FocalCodec-Stream: Real-time Low-Bitrate Speech Compression for Live Applications
Impact and Future Directions
This research represents a significant step forward in automated spoken language assessment. By offering a single-model, single-pass solution that aligns with human assessment practices, it simplifies the assessment pipeline, reduces model complexity, and mitigates error propagation. The resulting compact and deployable grader is well-suited for CALL applications, promising more reliable and efficient feedback for language learners.
Future work will explore extending this framework to cross-task generalization and addressing fairness considerations, further enhancing the utility and applicability of this advanced SLA system.
