TLDR: StressTransfer is a novel speech-to-speech translation (S2ST) system that accurately preserves word-level emphasis from the source language into the target language. It addresses data scarcity through an automated pipeline (EmphST-Instruct) that generates emphasis-aligned training data using LLMs. The system translates source speech into text with explicit emphasis markers, which then guide a controllable text-to-speech model to synthesize expressive speech. Evaluations show StressTransfer significantly outperforms baselines in maintaining emphasis while also achieving high translation quality and naturalness, highlighting the importance of prosody in cross-lingual communication.
In an increasingly interconnected world, speech-to-speech translation (S2ST) is a pivotal technology for breaking down language barriers. While early systems focused primarily on translating the literal meaning of words, effective communication relies heavily on subtle cues like emphatic stress – the way we emphasize certain words to convey focus, intent, or emotion. Neglecting these ‘paralinguistic’ details can lead to misunderstandings and a loss of expressive impact.
A new research paper introduces StressTransfer, an innovative S2ST system designed to preserve this crucial word-level emphasis across different languages. Developed by researchers Xi Chen, Yuchen Song, and Satoshi Nakamura, this system leverages the power of Large Language Models (LLMs) and advanced Text-to-Speech (TTS) technology to ensure that the nuance of spoken language is not lost in translation.
The Challenge of Expressive Translation
The field of S2ST has evolved significantly, moving from traditional multi-step processes (like Automatic Speech Recognition followed by Machine Translation) to more integrated, end-to-end models. Modern systems often incorporate powerful pre-trained components like Whisper for speech encoding and LLMs for text decoding, leading to robust semantic translation. However, a significant gap has remained in preserving expressivity, particularly explicit lexical or sentential emphasis. Previous attempts highlighted a major hurdle: the scarcity of large-scale, manually annotated datasets needed to train such sophisticated systems.
StressTransfer’s Innovative Approach
StressTransfer tackles this data bottleneck head-on with a multi-pronged strategy:
- EmphST-Instruct: To overcome the lack of training data, the researchers developed an automated pipeline called EmphST-Instruct. This system uses LLMs to generate vast amounts of high-quality, emphasis-aligned parallel corpora. Essentially, it translates source language text with stress annotations into target language text, ensuring both semantic accuracy and natural stress placement. This process involves multiple LLMs acting as ‘translation experts’ and another LLM as a ‘selection expert’ to pick the best translation, making the data generation scalable and cost-effective.
- EmphST-Bench: For rigorous evaluation, the team also introduced EmphST-Bench, the first benchmark specifically designed to assess emphasis preservation in speech-to-text translation. This benchmark features diverse stress patterns and uses both automatic metrics and human expert verification.
- Emphasis-Preserving S2TT Model: At the core of StressTransfer is an end-to-end Speech-to-Text Translation (S2TT) model. This model takes source audio and directly outputs target language sentences interleaved with explicit emphasis markers (e.g., **word**). It comprises a speech encoder (Whisper-large-v3), an adaptor to bridge the gap to the LLM, and a fine-tuned LLM (Qwen-2.5-3B) that generates the tagged text.
- Controllable TTS Module: The system integrates this emphasis-aware S2TT model with a controllable Text-to-Speech (TTS) synthesizer, CosyVoice2. This TTS module interprets the emphasis tags from the S2TT output and renders them as natural prosodic prominence in the synthesized target speech, adjusting elements like pitch, energy, and duration.
How it Works in Practice
Imagine speaking a sentence in English, emphasizing a particular word. StressTransfer first processes your speech, identifying the stressed word. It then translates the sentence into, say, Chinese, and inserts a special tag around the translated stressed word. Finally, a sophisticated text-to-speech system reads out the Chinese translation, using that tag to ensure the corresponding word is spoken with the correct emphasis, just as you intended in the original English.
Impressive Results
Comprehensive experiments demonstrated that StressTransfer significantly outperforms existing baselines, including powerful proprietary LLMs like GPT-4o and Gemini-2.5-Pro, in preserving expressive stress. On the EmphST-Bench benchmark, StressTransfer achieved superior Sentence Stress Reasoning Accuracy (SSR) scores. Crucially, it maintained competitive semantic translation quality (measured by BLEU and COMET scores) on standard datasets like CoV oST-2.
Subjective human evaluations of the synthesized speech further confirmed StressTransfer’s effectiveness, showing that it generates more emphasis-preserving translated speech and achieves higher audio quality compared to other leading models. An ablation study also confirmed the critical role of the EmphST-Instruct dataset in enabling stress-aware translation and validated the reliability of using an ‘LLM-as-Judge’ for evaluation.
Also Read:
- Enhancing Speech Recognition for Language Learners: A Focus on Proficiency
- ParsVoice: Unlocking High-Quality Text-to-Speech for the Persian Language
Paving the Way for Natural Communication
StressTransfer represents a significant step forward in speech-to-speech translation. By effectively preserving paralinguistic cues like emphatic stress, it moves us closer to truly natural and nuanced cross-lingual communication. This work highlights the often-underestimated importance of prosody in translation and provides a robust, data-efficient solution that sets a new baseline for future research in expressive speech translation.
